Every observability agent at this hackathon waits to be told something is wrong. It reads a firing alert and summarises it. We started from the opposite end: what does nobody get paged about?
For a streaming platform, that question has teeth. A DRM service denying 0.23% of licence requests will never trip an alert — but that's one viewer in 435 who paid for content and got an error, roughly 16,000 a day. Subtitles failing on a service nobody wrote an SLO for isn't a metric, it's an accessibility failure. These are the problems that lose subscribers quietly.
The core bet: published LLM-agent root-cause accuracy sits around 11% on OpenRCA. So we don't let the model do the detection. Every ratio, threshold and window is computed in Python from raw PromQL, following the Google SRE Workbook's multi-window burn-rate table. The agent receives a structured candidate and reasons about meaning — it never touches the arithmetic, and a test proves a hallucinated burn rate physically cannot reach the report.
What the live run taught us. Running it against real Grafana Cloud found five bugs no offline test could: list_datasources returns an envelope key we didn't handle; the stack has three Loki datasources and we were picking alert-state-history, which would have left log correlation silently empty; query_prometheus requires endTime even for instant queries — that one would have killed every query in the pipeline. We also caught ourselves reporting "0% error budget remaining" from 45 minutes of history. It withholds the number now and says why.
The honest edges. Free-tier Gemini allows five requests a minute; one agentic investigation burns that in a handful of tool-use turns, so we added backoff honouring the API's own retry delay and a per-investigation tool budget. And when a model returned a dict where our schema said string, we learned that a structured schema is a request, not a guarantee.
266 tests, 13 against a real mcp-grafana binary. The detection layer runs offline with no tokens at all.
Log in or sign up for Devpost to join the conversation.