StreamGuard AI — Agentic Reliability Supervisor

StreamGuard AI Architecture — Gemini + Google ADK with Grafana and Confluent MCP
StreamGuard AI — Live Multi-Source Streaming Investigation
StreamGuard AI — Evidence-Based Investigation Report
StreamGuard AI — Root Cause Analysis and Recommended Actions
Grafana Agent Observability — StreamGuard AI Generation Trace
Grafana Agent Observability — StreamGuard AI Agent Performance
Google Cloud Agent Platform — Deployed StreamGuard AI Agent
Google Cloud Agent Registry — Grafana MCP Server
Google Cloud Agent Registry — Confluent Kafka MCP Server
Grafana Cloud — StreamGuard AI Streaming Health Dashboard
Grafana Write-Back — StreamGuard AI Investigation Recorded in Grafana
Grafana Agent Observability — Detailed StreamGuard AI Traces
Confluent Cloud — Kafka Event Streaming Infrastructure
Grafana Cloud — StreamGuard AI Service Metrics and Runtime Telemetry

💡 Inspiration

During a major movie premiere or high-traffic streaming event, streaming infrastructure pipelines operate under massive load across ingest nodes, transcoding clusters, origin servers, and regional CDN delivery. When a stream begins buffering or dropping frames, site reliability engineers (SREs) and technical directors face dozens of disconnected monitoring dashboards, floodgates of Loki logs, Prometheus metrics, and Kafka event alerts.

Manual incident triage requires engineers to correlate raw telemetry across multiple monitoring systems, slowing investigation and response during critical broadcast windows. The real operational question during a premiere isn't just "Is CPU high?" — it is:

"What is failing? Where is it failing? What evidence proves it? How many viewers are affected? And what should the operator do next?"

We built StreamGuard AI to replace stressful manual dashboard hunting with an autonomous agentic workflow that correlates evidence across 2 operational MCP sources — Grafana and Confluent — to produce an evidence-backed streaming reliability investigation.

🎬 What it Does

StreamGuard AI is an autonomous reliability supervisor for cinema, OTT, and live video streaming, capable of overseeing live events with over 1.45M concurrent viewers. When an incident occurs or an operator asks to investigate streaming health:

Multi-Agent Delegation: The Google ADK Root Agent delegates tasks to specialized subagents:
- Broadcast Monitoring Agent: Queries 12+ Prometheus time-series metrics, Loki logs, and Grafana incidents via the official Grafana Cloud MCP Server (grafana/mcp-grafana).
- Event Streaming Agent: Connects via Confluent Kafka MCP to inspect topic schemas and consume real-time events across 4 mission-critical topics (stream-health, continuity-alerts, broadcast-incidents, streamguard-events).
Multi-Source Signal Correlation: Autonomously traces degradation paths (e.g., Ingest CPU Saturation at 94.2% ➔ Transcode Queue Depth over 140 frames ➔ Dropped Frames ➔ Viewer Buffer Overrun 4 minutes before viewer impact).
3-Tier Evidence-Based Reasoning: Strictly separates verified facts from derived observations and unconfirmed hypotheses.
Regional Correlation Safety: Validates whether telemetry contains explicit region_id tags before making geographical claims, eliminating AI hallucinations.
Closed-Loop Grafana Write-Back: Calls annotate_grafana_dashboard to record incident context directly onto live SRE Grafana dashboards (#ANN-8924).
AI Agent Observability: Uses Grafana Agent Observability to track the AI agent's own model calls, execution latency (<2.8s), token counts (~1,850 tokens), and execution cost (<$0.003 per investigation).

🛠️ How We Built It

Google Agent Development Kit (ADK) & Gemini 2.5/3.6: Powers multi-agent orchestration, tool routing, and deep cross-system reasoning.
Official Grafana Cloud MCP Server (grafana/mcp-grafana): Executes real-time PromQL metrics querying (query_prometheus), LogQL log analysis (query_loki_logs), Incident management, and dashboard annotations.
Confluent Kafka MCP Server: Inspects schema registries and consumes Kafka stream-health event messages across distributed topics.
Google Cloud Run: Serverless container runtime hosting the FastAPI backend and Director Console UI in us-central1 with auto-scaling down to zero.
Google Cloud Secret Manager: Safely injects Grafana and Confluent credentials into the runtime without committing secrets to source control.
Grafana Agent Observability (OpenTelemetry): Exposes real-time token tracking, latency, and cost telemetry for the AI agent itself.

🧗 Challenges We Ran Into

Cross-System Signal Correlation: Correlating time-series Prometheus metrics with unstructured Loki logs and event-driven Kafka messages required careful prompt orchestration and structured schema modeling to prevent hallucinated causal links.
Preventing Regional Hallucinations: In early tests, models tended to assume an incident belonged to a specific region based on incident titles alone. We implemented a strict Regional Correlation Safety rule that requires telemetry to explicitly contain validated region_id tags.
Sub-Minute Latency Under Load: Ensuring the multi-agent investigation pipeline finishes in under 45 seconds required parallelizing Grafana PromQL/LogQL queries alongside Confluent Kafka consumer polling.

🏆 Accomplishments That We're Proud Of

97% Faster Triage: Reduced MTTR (Mean Time to Resolution) discovery from 35 minutes of manual dashboard cross-referencing to under 45 seconds.
Proactive 4-Minute Buffer Warning: Diagnosed transcode CPU bottlenecks early enough to protect an estimated 260,000 viewers from experiencing buffering during peak load.
True Multi-Source Evidence Reasoning: Autonomous correlation of Grafana Cloud observability data with Confluent Kafka event streams without human query writing.
Observable AI at Negligible Cost: Full end-to-end AI agent observability in Grafana Cloud with complete token tracking at <$0.003 per investigation trace.
Live Cloud Run Deployment: Fully containerized, production-tested, and deployed live on Google Cloud Run.

📚 What We Learned

How to architect specialized subagents using the Google Agent Development Kit (ADK) to prevent prompt clutter and isolate observability concerns.
The power of the Model Context Protocol (MCP) for seamless, standardized communication between LLMs, Grafana Cloud, and Confluent Kafka.
Best practices for building human-in-the-loop AI reliability assistants that augment engineering decision-making rather than executing risky, uncontrolled production changes.

🚀 What's Next for StreamGuard AI

Automated Pod Autoscaling: Directly triggering Cloud Run worker pod scale-outs from 8 to 24 instances when transcode queues breach critical thresholds.
CDN Edge Synthetic Probing: Integrating synthetic playback probes across regional ISP edges for sub-second latency detection.
Voice-Activated Technical Director: Allowing broadcast directors to interact with StreamGuard AI hands-free via real-time voice queries during live productions.