Incident Film Crew

A deterministic crew of Gemini agents that investigates streaming incidents end-to-end via Grafana Cloud MCP — evidence, report, action plan — before the audience walks out.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

Inspiration

When a live premiere buffers, streaming platforms bleed viewers by the minute — and on-call engineers lose the critical first hour hopping between dashboards, logs, and alerts just to describe the problem.

What it does

Incident Film Crew is a multi-agent SRE assistant for streaming platforms. Report an incident and a deterministic pipeline runs: 🕵️ the Detective investigates live Grafana Cloud telemetry through the official Grafana MCP server (80+ tools — PromQL, LogQL, dashboards, alert rules); ✍️ the Screenwriter produces a SEV-rated incident report with timeline, blast radius and root cause; 🔧 the Stunt Coordinator delivers ranked mitigations, permanent fixes, and the exact Grafana alert rules that would have caught it earlier — ending with a "showtime status": can the premiere continue?

How we built it

  • Google ADK + Gemini on Vertex AI: LlmAgent Director orchestrates a SequentialAgent pipeline — deterministic multi-step flow with state passed via output keys (evidence → incident_report → action_plan)
  • Grafana Labs integration at runtime: official grafana/mcp-grafana MCP server loaded as an ADK McpToolset (stdio), querying a live Grafana Cloud stack
  • Demo world: a telemetry seeder simulates "PopcornFlix" — a CDN config rollout wrecks the Meridian Falls live premiere in eu-west (metrics via Influx line protocol → Mimir, logs → Loki)
  • Deployed on Cloud Run, container bundles the MCP server binary

Challenges

Free-tier Gemini quotas (20 req/day) can't power agentic workloads — one investigation is ~30 LLM calls; we moved to Vertex AI and added retry backoff. The mcp 2.x package broke ADK imports (pinned <2). Windows console encoding vs emoji-rich agents was its own adventure.

What we learned

Deterministic beats clever: an explicit SequentialAgent pipeline is more reliable and more explainable to SRE teams than free-form delegation. MCP is a genuine abstraction win — 80+ observability tools with zero custom API glue.

What's next

Grafana OnCall/Incident integration for real paging, bidirectional alert-rule creation from the Stunt Coordinator's plan, and multi-incident memory.

What we learned

What's next for  Incident Film Crew

Built With

Share this project:

Updates

Submission history