Aegis — The Release Gate for AI Agents

Inspiration

I run AI agents in production and I'm the entire on-call rotation. Every time I promote a new prompt or model I hit the same gap: CI is green, the unit tests pass, and the agent still regresses under real traffic — latency creeps, tool calls fail, exceptions spike. There is no release gate that understands agent runtime behavior. So I built one.

What it does

Aegis is a pre-release gate for AI agents. When you propose a release, a Gemini 3 agent reads the candidate's own production traces from Arize Phoenix, reasons over the actual tool-call spans — arguments, ordering, errors — and renders a structured GO / BLOCK verdict, with every claim linked to the exact trace evidence. Then it hands control to a human: you approve or override, and your decision (who, why, against which evidence) is written back to Phoenix as a permanent, tagged audit record.

In the demo it catches a concurrency race condition: fetch_cart_data firing before authenticate completes in 5 of 15 checkout traces — a 33% failure rate throwing UnauthorizedSessionException. That class of bug lives in span timing and ordering, so text-only evals miss it entirely.

How we built it

• Gemini 3.5 Flash on Vertex AI (global endpoint) as the sole reasoning layer.
• Google Agent Development Kit (ADK) for the agent loop, deployed on Cloud Run as a no-login web app (FastAPI + a custom single-page dashboard that streams the agent's steps over Server-Sent Events).
• The Arize Phoenix MCP server as the load-bearing integration: the agent calls list-traces / get-spans to read evidence, and upsert-prompt / add-prompt-version-tag to write the audit record.
• Secrets in Secret Manager; the human approval is a separate gated endpoint, so no write happens until a person decides.

Challenges we ran into

• Gemini 3 isn't served on regional Vertex endpoints — us-central1 returned 404 while 2.5 worked. The fix was switching to the global location.
• A passing eval script hid a broken browser path. The live verdict card wouldn't render until the SSE layer was mapped to ADK's real event schema (function_call / function_response / text parts). Lesson: test the path the user actually touches.
• Making the demo truly no-login meant overriding an org Domain-Restricted-Sharing policy so Cloud Run could serve allUsers.
• The MCP has no "run experiment" tool, so we scoped the loop tightly to read-evidence → reason → human-gated write.

Accomplishments that we're proud of

• A genuinely beyond-chat agent: it plans, reads real telemetry, reasons to a structured verdict, and finishes the job with a gated write — not a chatbot.
• The MCP is load-bearing — remove it and there is no product. Gemini does the actual reasoning over the failing spans, not narration.
• The human stays the release authority, and the override is recorded: accountability, not automation theater.

What we learned

• Agent observability is its own discipline — the failures that matter (tool ordering, race conditions) live in span attributes, not message text.
• Keeping a human in the loop, with the machine's reasoning as linked evidence and the human's decision recorded, beats full automation for high-stakes release calls.

What's next for Aegis — The Release Gate for AI Agents

• Author-and-prove-the-fix: have the agent propose a corrected orchestration and validate it with a real Phoenix experiment before promotion.
• More evidence sources: latency/error forecasting and cross-run regression deltas.
• Policy packs so teams can codify what "block" means for their own agents.

Built With

• arize-phoenix
• docker
• fastapi
• gemini
• google-adk
• google-cloud-run
• model-context-protocol
• python
• secret-manager
• server-sent-events
• vertex-ai