AgentCI

CI for AI agents: catches silent prompt regressions, autonomously root-causes them via Phoenix MCP, proves the fix, and permanently learns from every failure.

Comment

Inspiration

Everyone is shipping AI agents, but almost no one can answer: "I changed my agent's prompt — did I make it better, or silently break something?" For traditional code we solved this with regression CI: change code → tests run → a red build blocks the merge. For agents, that safety net doesn't exist. A one-line prompt edit like "be more concise" can quietly drop a refund-eligibility rule — the agent still answers fluently and confidently, it's just wrong, and a human skimming five outputs would approve it.

What it does

AgentCI is regression CI for AI agents. On a candidate prompt change to an instrumented support agent, it:

  1. Detects — runs the candidate over a frozen eval suite as a Phoenix experiment and flags pass→fail flips vs. baseline. A red gate blocks the change.
  2. Investigates — a Gemini Regression Investigator agent autonomously root-causes the failure through the Phoenix MCP server: it forms a hypothesis, chooses its own MCP queries over the experiments and traces (a genuine reason-act loop, not a script), and names the root cause in plain English.
  3. Learns — the investigator authors a guard (a deterministic assertion and/or scoped LLM-judge rubric) that catches that exact failure. Guards are admitted only via a two-sided discrimination test: they must FAIL the regressed answer and PASS the gold one.
  4. Fixes & proves — a separate fix-author agent proposes a prompt fix, which must prove held-out lift on a frozen cross-family ruler (a non-Gemini judge — the agent can't grade its own homework).
  5. Asks a human — promotion and guard-minting happen only on explicit human approval via the dashboard. Nothing auto-merges.
  6. Remembers forever — approved regressions go into a git-tracked Quality Memory; the next time a similar failure appears (matched by policy), the investigator starts with the prior lesson. Tripping a previously-learned guard is an instant red — no investigation needed. Every regression caught makes the suite permanently stronger. That's the self-improvement loop.

Try it live: the hosted dashboard replays a real recorded run, and the "Run this investigation live" button fires the real Gemini investigator against the Phoenix MCP server in real time — fresh root cause, real MCP-call count, no replay.

How we built it

  • Target agent: a config-driven Google ADK support agent (Gemini), instrumented with phoenix.otel.register(auto_instrument=True) → OpenInference traces in Phoenix Cloud.
  • Investigator: an ADK LlmAgent with @arizeai/phoenix-mcp mounted as a native McpToolset — the agent introspects its own observability data at runtime through MCP, not a human in a separate harness. Diagnosis and fix-authoring are deliberately separate agents.
  • Eval harness: a 40-ticket SaaS-billing suite with a fixed 24/16 tune/held-out split, scored by four LLM-as-judge dimensions (correctness, groundedness, completeness, policy-reference) via Phoenix experiments.
  • Determinism: everything runs at temperature 0 with pinned model IDs, and every LLM/judge/MCP call goes through a record/replay cache — 82 tests run fully offline, and the demo replays a captured real run.
  • Surface: a CLI (agentci check) + FastAPI dashboard on Cloud Run, built around one demo beat: the fluent-wrong answer beside the correct one.

Data sources

Synthetic but realistic: 40 SaaS-billing support tickets with gold resolutions and a knowledge base of billing policies (refund windows, proration, cancellation), generated once and frozen so every run is comparable. All run artifacts (experiments, traces, datasets) live in Phoenix Cloud.

Challenges we ran into

  • Temperature 0 is not determinism. Re-running the unchanged baseline prompt still produced 2/16 pass→fail flips — judges wobble, knife-edge cases sit exactly at threshold. We measured this sampling-noise floor empirically (scripts/calibrate_gate.py) and calibrated the gate against it, instead of pretending zero-flip was achievable.
  • A recovery is not an improvement. Our first promotion bar demanded +0.05 held-out lift — which structurally rejects every real fix, because a regression recovery's best case is parity with baseline. The gate now encodes "no worse than baseline, no new held-out regressions."
  • Self-grading is a trap. Letting the fix-author's model family judge its own fix inflates lift; we froze a cross-family ruler.

What we learned

The biggest lesson: making non-deterministic agent behavior testable is mostly an engineering discipline problem — pinned models, a record/replay cache as a first-class citizen, and gates calibrated against measured noise rather than wishful zero. And "meaningful MCP use" means giving the agent the same observability access an SRE has, then letting it choose its own queries — the recorded trajectories show genuinely different investigation paths for different failure types.

What's next

GitHub-PR integration (gate as a check run), multi-domain eval suites, and guard generalization — letting the investigator promote a case-specific guard into a policy-level invariant after it has caught the same class of failure twice.

Built With

  • cloud-run
  • fastapi
  • gemini
  • google-adk
  • openinference/opentelemetry
  • phoenix-(arize)-+-phoenix-mcp-server
  • python/uv
Share this project:

Updates