AI Guardian

The Problem

Traditional monitoring generates alert fatigue — too many false positives, no context about why something is failing, and no correlation between related issues. Operators waste time investigating noise instead of real problems.

The Solution

AI Guardian combines statistical anomaly detection with LLM-powered analysis to provide intelligent, contextual alerts. It understands the infrastructure topology, correlates events across services, and explains what’s happening in plain language.

Architecture

• FastAPI server with APScheduler for periodic collection
• 5 collectors: Prometheus metrics, Loki logs, CrowdSec security events, container health, service availability
• Statistical engine: Z-score and IQR-based anomaly detection with adaptive baselines
• LLM analysis: Mistral 7B (via Ollama/LocalAI) for root cause analysis
• SQLite state: Persistent alert history, deduplication tracking, escalation state

Detection Pipeline

Collectors → Normalize → Statistical Analysis → Anomaly Score
                                                      ↓
                                              LLM Root Cause Analysis
                                                      ↓
                                              Alert (if score > threshold)
                                                      ↓
                                              Dedup → Cooldown → Escalation

Intelligent Features

• Deduplication: Same alert won’t fire repeatedly within cooldown window
• Escalation: If an issue persists across multiple check cycles, escalate severity
• Correlation: Groups related anomalies (e.g., high CPU + container restart + error log spike)
• LLM context: Provides human-readable explanation of what likely caused the anomaly
• Ignore patterns: Configurable filters to suppress known noise (ephemeral containers, CI runners)

Grafana Integration

Custom Grafana dashboard using the Infinity datasource plugin:

• Real-time alert status
• Anomaly score trends
• Collector health
• Alert history timeline
• Direct links to relevant Grafana panels for each anomaly type

Results

• Reduced alert noise by ~80% compared to static threshold monitoring
• LLM analysis provides actionable context, reducing mean-time-to-diagnosis
• Zero missed critical incidents since deployment
• Adaptive baselines handle workload changes without manual threshold tuning