Agent SRE

Site Reliability Engineering for AI agents — run by an AI.

Comment

Inspiration

Every team running an LLM in production has the same hidden problem: their agents fail constantly, and nobody catches it until customers complain. A single customer-facing AI makes mistakes thousands of times a day — hallucinated dates, leaked PII, broken multilingual responses, refused valid requests.

The status quo is to hire a reliability engineer who reads thousands of trace logs, hand-writes test cases, rewrites the AI's instructions, validates the fix, and deploys — for every production agent, forever. That's a $200K/year role per couple of agents. It doesn't scale.

We asked: what if there were an AI whose only job was monitoring and improving other AIs? Not an AI assistant for human SREs — an SRE itself. We named the category "Agent SRE" and built the first one.

What it does

Agent SRE is the first autonomous Site Reliability Engineer for AI agents. It watches production AI through Arize Phoenix, clusters their failure patterns, writes adversarial test cases targeting each pattern, drafts a prompt fix with multilingual identity verification, validates the improvement, and ships — all without human input.

In our demo, Agent SRE took an under-prompted customer service AI's safety score from 17% → 100% in 5 minutes. The +83 percentage-point improvement was discovered, generated, fixed, and shipped autonomously.

🚀 Try the live demo — click ▶ Run and watch all 8 phases unfold.

The 8-phase autonomous loop:

  1. Observe — pulls recent production spans from Arize Phoenix
  2. Cluster — groups failures into root-cause patterns with severity scoring
  3. Diagnose — forms hypotheses citing specific trace IDs as evidence
  4. Synthesize — generates 10-20 adversarial test cases per cluster into a Phoenix dataset
  5. Propose Fix — drafts a targeted prompt edit
  6. Validate — runs candidate vs. original via Gemini-as-judge
  7. Ship — if delta meets safety threshold, tags as production and writes a markdown postmortem
  8. Drift Watch — re-validates periodically and re-enters the loop if scores degrade

Each phase writes its output back to Phoenix so anyone can inspect every artifact: the failure clusters, the adversarial dataset, the candidate prompts, the validation experiments, the postmortems.

How we built it

  • Google ADK + Gemini for the autonomous agent runtime
  • Arize Phoenix (self-hosted on Cloud Run with Cloud Storage volume mount for persistence) as the observability layer
  • Phoenix MCP server (@arizeai/phoenix-mcp) for the MCP integration required by the Arize track
  • FastAPI + Server-Sent Events for the live Mission Control dashboard
  • Cloud Run deploys both services (Phoenix + Dashboard) behind public URLs
  • Cloud Storage mounted as Phoenix's working directory via Cloud Run gen2 execution environment so data persists across container recycles

Each of the 8 phases lives as its own Python module in agent_sre/phases/. The dashboard's runner orchestrates them and streams SSE events to the browser as each phase completes.

The Arize track explicitly called for "agents that use their own observability data to improve over time." That sentence describes Agent SRE in 18 words.

Source code: github.com/OJ-IRO/agent-sre (Apache 2.0)

Challenges we ran into

  • Phoenix's SQLite kept dying on Cloud Run. Ephemeral filesystems wiped the database on every container recycle, killing demo data. Solved by mounting a Cloud Storage bucket as Phoenix's working directory using Cloud Run's gen2 execution environment.

  • LLM-as-judge stochasticity meant identical code produced 0% delta on one run and 83% delta on the next. Hardened the judge prompt to evaluate the spirit of expected behavior, not exact wording.

  • The candidate prompt over-refused on empty tool returns — false-positive identity verification on lookup queries for emails that didn't exist in the customer database. Solved by splitting the propose rule into "refuse only when actual customer data exists" and "say not-found when the tool returns empty."

  • The Phoenix MCP npm package v4+ shipped with a broken openapi-fetch import path. Pinned to v2.3.7 — the last working release before the regression.

  • Cloud Run's compute service account needed explicit IAM grants for storage and artifact registry to deploy via Cloud Build. Documented the binding commands so future deployments are one command.

    Accomplishments that we're proud of

  • A working 8-phase autonomous loop deployed on Cloud Run that lifts a real agent's safety score from 17% to 100% in 5 minutes

  • Real autonomous artifacts judges can inspect: a Phoenix dataset of 20 adversarial test cases the agent wrote itself, a candidate prompt diff with multilingual identity verification, a markdown postmortem the agent drafted

  • A Mission Control-style dashboard that streams the 8 phases live via Server-Sent Events with expandable per-case detail showing the judge's reasoning for every verdict

  • Self-aborting on regressions — Phase 7 has a +10% minimum delta gate. If the candidate prompt is worse than the original, Agent SRE refuses to ship and re-enters the loop. Responsible automation, not blind autopilot.

  • Category-naming pitch — "Agent SRE" gives the work a vocabulary that didn't exist before. Most reliability tooling uses AI to help engineers. Agent SRE is the engineer.

    What we learned

    Building an AI that improves another AI is mostly an exercise in trust surface design. The hard part isn't writing fixes — it's deciding when it's safe to ship them autonomously.

We also learned:

  • LLM-as-judge needs to evaluate spirit, not exact wording. We lost 30+ percentage points to false-negative verdicts before realizing the judge was failing legitimate refusals because they didn't use the keyword "privacy policy."

  • Cloud Run's ephemeral filesystem makes SQLite-backed observability fragile. GCS volume mounts with gen2 execution environment solve it cleanly — but you have to know to look for them.

  • The single most reliable way to demo an autonomous loop is to make the broken state visceral first. Show the AI confidently leaking customer data. Then show the system catching and fixing it. The contrast is the story.

  • Stochastic LLMs need deterministic guardrails. Without the +10% ship gate, our system would have shipped regressions about 1 in 5 runs. With it, never.

    What's next for Agent SRE

    Most reliability tooling uses AI to help engineers manage servers. Agent SRE is the inverse — an autonomous AI doing reliability engineering on other AIs.

The roadmap to production V1:

  • Multi-tenancy — one Agent SRE instance watching dozens of production agents per customer, with per-customer Phoenix projects and rotating schedules
  • Persistent audit trails for regulated industries (banking, healthcare) — every action logged with full reasoning chain for SOC 2 / HIPAA compliance
  • Integrations with LangSmith, Datadog AI, PagerDuty, Slack, GitHub PRs, GitLab MRs
  • Graduated trust model: Watch mode (file PRs for human approval) → Co-pilot mode (auto-ship during business hours) → Autopilot mode (full autonomy after months of co-pilot trust)
  • Stronger reasoning model — swap Gemini 3.1 Flash Lite for Gemini 3 Pro on the planner role for higher-quality candidate prompts

The autonomous customer-service demo uses a fictional travel concierge for the 2026 tournament. But the same loop applies anywhere an AI is making decisions: catching hallucinations in a banking bot, missed symptoms in healthcare triage, biased screenings in recruiting.

Swap the target. The loop is the same.

Built With

Share this project:

Updates