-
-
Failures
-
Scorecard
-
Patch Review
Inspiration (The $500 Hallucination)
Imagine a customer asks an enterprise support chatbot for a refund. The agent hallucinates the policy and wrongly approves a $500 refund, completely skipping the required database check. This happens in roughly 15% of complex edge cases, costing enterprises thousands of dollars and creating massive compliance risks. Currently, human engineers must manually sift through telemetry traces, write new evals, and deploy prompt patches—a slow, reactive process. I wanted to build an autonomous AI engineering teammate to close this loop.
What it does
AgentEval Autopilot is a self-improving LLMOps system built on a code-owned FastAPI runtime. Running entirely offline and asynchronously — so it never impacts live customer chat latency — a Quality Monitor meta-agent sweeps traces from a live support agent via Arize Phoenix.
When it detects failures, it uses the Model Context Protocol (MCP) to extract the traces, writes its own eval cases, and proposes a prompt/tool-policy patch. It then runs before-and-after tests across all three failure archetypes and presents a visual scorecard alongside a clear diff of the exact prompt changes to a human engineer for deployment approval. Nothing ships without human sign-off.
How I built it
I built the application using a code-owned Python/FastAPI backend to meet the Google Cloud Agent Builder track requirements while maintaining full custom control.
Reasoning Engine: I used the Google Gen AI SDK (google-genai) to power both the demo support agent and the quality monitor meta-agent, using Gemini 3 for all complex reasoning and patch generation. Observability: I instrumented every LLM call using OpenInference, flowing traces directly into Arize Phoenix. Tool Calling & MCP: The Support Agent uses deterministic Python tools (like lookup_refund_policy). The Quality Monitor agent integrates with the Phoenix MCP server to dynamically query traces, fetch spans, and inspect prompt versions during the evaluation cycle. Frontend: I built a custom dark-mode operational dashboard to visualize the 0% → 100% improvement scorecard and provide a human-in-the-loop approval interface.
Challenges I ran into
The biggest challenge was making the improvement loop believable, not just functional. It is trivially easy to build a system that "fixes" a single hand-crafted failure — any prompted LLM can do that. The hard part was designing three genuinely distinct failure archetypes (a missing tool call, a hallucinated policy, and an unsafe direct action) and having a single autonomous cycle diagnose and fix all three simultaneously with one unified prompt patch.
I also had to resist the temptation to show a suspiciously perfect 0% → 100% jump and instead focus on making the underlying mechanics transparent: every proposed rule change is shown as a visible diff, every failed check is listed with its evidence, and the risk score is surfaced so the human reviewer has full context. The goal was to build trust, not just show a number.
Accomplishments that I am proud of
I'm most proud of the fact that the system covers three fundamentally different failure modes — not just variations of the same bug. The entitlement scenario tests tool-use compliance, the refund scenario tests groundedness against real policy data, and the billing scenario tests safety escalation. Gemini analyzes all three failure patterns and generates a single, coherent prompt patch that addresses each root cause with a specific operational rule. That generalization — not overfitting to one test — is what makes this a platform rather than a script.
I'm also proud of the human-in-the-loop architecture. The approval gate with diff review, risk scoring, and one-click rollback ensures this powerful automation remains safe for enterprise production environments.
What I learned
I learned the immense power of the Model Context Protocol (MCP). By connecting Gemini directly to the Phoenix MCP server, I didn't have to write complex API polling scripts to fetch telemetry. The agent could simply ask Phoenix for the trace data it needed to diagnose hallucinations. I also deepened my understanding of OpenInference and how crucial deterministic evaluations are alongside LLM-as-a-judge patterns.
What's next for AgentEval Autopilot
The current demo validates the proposed patch against the same three failure scenarios. The immediate next step is a regression suite: running the patched prompt against a broader set of 50+ passing cases to mathematically prove the fix didn't break existing behavior. This is the standard practice in enterprise prompt engineering, and the architecture already supports it — the eval runner accepts any list of cases.
Beyond that, I plan to add A/B testing (deploying the new prompt to a shadow environment against live traffic before human approval), expanded evals for latency regression and toxicity detection, and cost-aware monitoring that triggers the improvement cycle only on detected anomalies rather than running Gemini continuously.
Log in or sign up for Devpost to join the conversation.