Vigil

Autonomous media-operations agent. Takes the page, finds the cause, ships the fix.

Inspiration

It is the 89th minute of the World Cup final. A penalty is about to be taken, two billion people are watching one stream, and the player-stats endpoint hits a cache stampede. P99 latency triples, regions start dropping, and PagerDuty fires. In the old world that is a 30-minute war-room scramble: someone gets paged, fumbles for a laptop, and stares at dashboards that show what broke but never why, while the moment everyone tuned in for slips away.

We did not want a chatbot that answers questions about the outage. We wanted the outage to be over before anyone reached for their phone: an agent that takes the page, finds the cause, and ships the fix on its own. And we wanted it grounded in the one system that actually holds the truth of a live broadcast, the telemetry firehose sitting in ClickHouse.

What it does

Vigil is an autonomous SRE for media operations. It takes a real production page (from PagerDuty, Grafana, a built-in ClickHouse watchdog, or a manual trigger) and walks the entire incident lifecycle as a four-agent pipeline:

1. Triage runs SQL over ClickHouse (viewer_events, cdn_edge_logs, service_spans) to confirm the anomaly is real, classify severity and blast radius, and check its own incident history for precedent.
2. Diagnostician iteratively queries ClickHouse to triangulate root cause across viewer telemetry, CDN edge logs, and service spans. It grounds the fix location against the real repository, and it only proceeds with evidence, or structurally abstains when it cannot.
3. Remediator writes the actual fix as a merge request with a tight diff on the connected repo (GitHub, GitLab, or Bitbucket), links the incident, and kicks off CI. Vigil never merges; a human always signs off.
4. Scribe posts a human-readable timeline to the on-call Slack channel, including an honest "ABSTAINED" headline when the Diagnostician could not answer safely.

Every step is a real action against real infrastructure. In our demo, a genuine cache stampede fired at a live streaming platform (PITCH) becomes a genuine merge request, "coalesce stats cache-miss queries behind a per-key lock", in under a minute, with no human in the loop. No one was paged. No one touched a keyboard.

How we built it

• Google Agent Development Kit (ADK) orchestrates the four sub-agents as a deterministic sequential pipeline, passing typed session state from one stage to the next (TriageReport to DiagnosisReport to RemediationReport to IncidentNarrative), so each agent's prompt is fed the previous agent's structured output.
• Gemini 3.6 Flash on Vertex AI is the reasoning model, behind a region router that keeps calls on the global endpoint where the GA model has the most quota headroom and the lowest tail latency.
• ClickHouse is the diagnosis substrate, reached through the official mcp-clickhouse server (run_query, list_tables, list_databases) over the Model Context Protocol, unmodified. All correlation happens in-prompt, one auditable SQL string at a time, with no black-box natural-language-to-SQL layer in between.
• A hosted customer proxy lets a tenant paste their own ClickHouse host, user, and password and have Vigil run the MCP surface for them, with every query checked against a read-only allowlist before it reaches the cluster. Teams already running their own mcp-clickhouse can point Vigil at it instead.
• The Remediator opens real merge requests via the code-host MCP and REST APIs against whichever repository the tenant connects, chosen live in the onboarding UI rather than hard-coded.
• Google Cloud Trace, via OpenInference, captures every reasoning step end to end using the same Application Default Credentials as Vertex AI, so there is no separate collector or API key.
• Multi-tenant by design: every request carries a tenant key, and tenants drive the agent against their own ClickHouse, repository, and Slack. A ClickHouse watchdog polls each tenant every 30 seconds and pages Vigil itself when telemetry crosses a threshold, so a fresh tenant gets real detection before wiring up any alerting.
• A Next.js frontend and a FastAPI plus WebSocket server stream the agent's live reasoning into a NOC-style dashboard. Everything runs on Cloud Run.

The safety story runs two layers deep. On the code-host side the agent literally cannot call destructive tools, because they are not in its toolset. That is an allowlist, not a prompt instruction. On the ClickHouse side the guarantee sits one layer lower: the agent connects as a dedicated vigil_agent role with SELECT-only grants, and we verified it by running an INSERT as that role and confirming ClickHouse rejects it at the database layer, independent of anything the prompt says.

Challenges we ran into

• The rules name a specific server, and it is not the one the blog posts describe. ClickHouse's newer ClickStack observability MCP reads almost exactly like what the brief asks for, but the rules name the older, thinner mcp-clickhouse (three tools, no hypothesis assistant). We nearly built the whole integration against the wrong product's tool vocabulary before reading the actual repository and confirming the real surface.
• A major-version split in the mcp Python SDK broke the connection with no useful error. The newest mcp-clickhouse pulls fastmcp 4.x, which needs mcp>=2.0, while our google-adk chain had resolved mcp 1.27. The failure was a generic "Session terminated" with no traceback on either side, which sent us cross-testing every client and server combination before we found the real cause was unrelated to versions.
• The real bug was a Host-header allowlist, two layers away. mcp-clickhouse's DNS-rebinding protection rejects any request whose Host header is not on an explicit allowlist, and a bare localhost pattern does not match localhost:8000 because the header carries the port. With auth on it surfaced as that same opaque "Session terminated"; with auth off it became a clean 421 Misdirected Request. The fix is a host wildcard pattern documented nowhere we could find, and we only got it by reading the middleware source.
• A query that "succeeds" with zero rows is more dangerous than one that throws. Our continuous-watch parser was written against a {"records": [...]} shape, but mcp-clickhouse actually returns {"columns": [...], "rows": [[...]]}. It failed silently, so the watchdog polled ClickHouse and returned nothing on every tick regardless of the data. We only caught it while proving schema discovery against a table the agent had never seen, which made the silent zero obvious for the first time.
• A media-operations agent needed a real schema, not a generic one. It would have been easy to ship infra-generic vocabulary (services, hosts, exceptions) pointed at ClickHouse. We instead designed the schema and the diagnostic prompts around signals a live broadcast actually has: rebuffer rate, CDN POP, ad-break fill, bitrate collapse. The judging criteria reward genuine understanding of the problem space, and a generic tool does not show that.

Accomplishments that we're proud of

• It closes the loop for real: alert, SQL diagnosis, a merge-ready MR with a correct fix, Slack post, with zero human keystrokes.
• It runs against a judge's own account, ClickHouse, and repository, not a canned replay. Multi-tenant, bring-your-own-credentials, bring-your-own-MCP.
• The official partner MCP server is wired into a working multi-agent loop unmodified, imported and called at runtime, not just named in a README.
• The security boundary is verified, not asserted. We tried to break it, and ClickHouse itself rejected the write.
• It abstains honestly when it is not sure, which is the hardest and most important behavior for an autonomous agent that touches production. Two of our six eval scenarios are deliberately unresolvable from telemetry alone, and the rubric rewards abstaining over guessing.

What we learned

• Thin tools force better agents. Not having a magic NL-to-SQL layer made the diagnosis loop transparent and debuggable, because every step is a SQL query a human can audit.
• A generic error message is a signal to keep digging, not a root cause. "Session terminated" looked like a version problem for a while; it was a header-matching bug two layers away.
• A silent zero hides better than a loud failure. We are now suspicious of any code path whose failure mode is "returns nothing" rather than "raises".
• Autonomy is mostly about knowing when to stop. The abstain path and the structural safety gates did more for trustworthiness than any single clever prompt.

What's next for Vigil

• Provision-on-onboard: have Vigil stand up the ClickHouse Cloud service and MCP proxy for the customer, removing the "bring your own exact schema" burden entirely.
• Richer remediations: multi-file fixes, config changes, and rollbacks, not just single-file diffs.
• Learning from outcomes: feed merged or reverted MRs back into Triage's incident-history reflection so Vigil improves per tenant, incident over incident.
• Proactive mode: move from paging on threshold breaches to predicting stampedes before they cascade.
• Beyond media: general production incident response on any ClickHouse-backed observability stack.

Built With

Python, Google Agent Development Kit, Vertex AI, Gemini 3.6 Flash, ClickHouse, mcp-clickhouse, Model Context Protocol, Google Cloud Run, Google Cloud Trace, OpenInference, FastAPI, WebSockets, Next.js, Supabase, GitLab, GitHub, Bitbucket, Slack, OpenTelemetry.

Built With

• bitbucket-api
• clickhouse
• fastapi
• gemini
• github-api
• gitlab-api
• google-agent-development-kit
• google-cloud-run
• google-cloud-trace
• mcp
• nextjs
• opentelemetry
• postgresql
• python
• react
• slack-api
• supabase
• tailwindcss
• typescript
• uvicorn
• vertex-ai