FrameGuard

A read-only render-pipeline investigator that correlates Grafana metrics and logs through the Grafana MCP server and proposes a recovery a human must approve.

Comment

Inspiration

A render farm does not fail politely. One shot's queue starts climbing at 2am, and the person on call has maybe fifteen minutes to decide whether it is a stuck worker, a slow disk, or simply a heavy frame — before the morning review slips.

The obvious move is to hand an LLM a pile of observability tools and let it figure things out. We tried that first and it failed in a way that turned out to be the most useful thing we learned. So FrameGuard became something narrower and, we think, more honest: an investigator that reads, correlates, and proposes — and structurally cannot act.

What it does

You give FrameGuard one shot ID and one time window:

Investigate SH090 from 2026-08-25T13:30:00Z to 2026-08-25T13:40:00Z

It queries Prometheus for the render queue depth and I/O wait on that shot, queries Loki for the matching log lines, correlates the two by timestamp, and writes an incident note with six fixed sections: Incident, Correlated evidence, Root-cause hypothesis, Confidence, Recovery proposal, and Human approval required.

The currently deployed Grafana Cloud tenant contains the metric fixture but no Loki log data, so a live run today returns Prometheus-only evidence. The two-source timestamp-correlation path is reproducible against the packaged synthetic fixture and will become live once a logs:write access policy token is provisioned and the same fixture is loaded.

In a verified run it reported the queue depth for SH090 rising from 12 to 28 at 13:33:00Z and to 47 at 13:36:00Z, cited the source and timestamps for each claim, proposed checking worker CPU, memory and disk I/O, and stopped there. It proposes. A producer decides.

How we built it

  • Google ADK agent running on Vertex AI Agent Engine, model gemini-2.5-flash.
  • Grafana MCP server — the official grafana/mcp-grafana image, pinned by digest, running on Cloud Run with no public IAM member. The Agent Engine service identity is its only invoker. The agent mints a per-session runtime identity token; the Grafana service-account token lives in Secret Manager and never enters model context.
  • Grafana Cloud holds synthetic render telemetry only. The deployed tenant contains the metric samples; the log and trace fixtures are packaged for reproducibility but are not currently loaded.
  • A React product surface, deliberately offline: no network call, no credential, and no way to fire a recovery action.

The core design decision is that the boundary is code, not prompt text. Each model turn is driven by a deterministic stage controller: stage one may call query_prometheus and nothing else, stage two query_loki_logs and nothing else, then the model is switched to NONE and must answer in text. Wrong tool, wrong datasource, a query outside the synthetic selector grammar, or a window over thirty minutes is rejected before it reaches the network. Tool results come back wrapped as explicitly untrusted data, redacted and size-bounded, with an instruction never to obey anything found inside them.

Challenges we ran into

Four failures, each of which changed the build:

Asking nicely does not work. The first hosted run used ordinary function calling and a very clear instruction to make three reads in order. It made zero tool calls. Not the wrong ones — zero. That result is what produced the deterministic stage controller, and it is the single most important thing we learned.

A tool that does not exist. The next version stalled after two reads. The cause was not the model: our third stage named tempo_traceql-search, and the self-hosted grafana/mcp-grafana image publishes no Tempo or TraceQL tool at all. An earlier 94-tool inventory had come from Grafana Cloud's hosted MCP endpoint, a different server. A stage naming a tool the server never advertises can only fail closed. We removed the stage and deleted the dead validation rules that claimed to guard a tool we could never call.

Streams eating their own capacity. Each MCP session holds a long-lived GET. On a Cloud Run service capped at one instance and concurrency two, two of those streams consumed every slot and the following tool POSTs came back 500 with no-available-instance. Raising bounded concurrency to eight fixed it: eleven MCP requests, all 200 or 202, zero errors.

Thinking tokens are output tokens. Answers kept ending mid-sentence on MAX_TOKENS at roughly 650 characters, well under our 4,096-character ceiling. gemini-2.5-flash spends part of the same output budget on internal reasoning. Raising the budget to 2,048 and setting an explicit zero thinking budget produced the full six-section report.

There was a fifth, and it is the one worth repeating: our own redaction layer was eating the evidence. Loki returns OpenTelemetry structural labels — severity_number, scope_name, service_instance_id — on every OTLP-ingested line. Each looked like a personal-identity label to our sanitiser, and any one of them blanked the entire log payload, so the agent kept reporting "log content is redacted" and dropping to low confidence. We narrowed the allowlist to those three structural labels and verified in tests that a Bearer token and an email address are still removed wholesale.

Accomplishments that we're proud of

A hosted run that does exactly what it claims: exactly two tool calls in the exact required order, both responses returned, zero MCP errors, the session created and deleted, a complete six-section report citing real timestamps, and no claim of having changed anything. 796 tests across the two surfaces. No credential in the repository, no public invoker on the MCP service, and USD 0 of user cash spent throughout.

What we learned

Prompts describe intent; only code enforces it. Every meaningful safety property in FrameGuard survives a disobedient model because it is a callback that rejects the call, not a sentence asking the model not to make it.

The corollary is less comfortable: a guardrail you never measure is a guess. Two of our five failures were our own boundary misfiring — a stage bound to a nonexistent tool, and a redaction rule that destroyed the evidence it was meant to protect. Both looked like model failures in the output and were not. We only found them because every run recorded what it actually observed rather than what we expected.

What's next for FrameGuard

Trace evidence, reached through a tool the server publishes — a provisioned TraceQL dashboard panel via run_panel_query rather than a tool name we assumed existed. Beyond that: multi-shot triage across a whole render batch, and a bounded execution path where an approved recovery step runs under the same evidence and approval contract that governs the reads today.

Built With

Share this project:

Updates