TraceFix

Inspiration

We kept seeing the same gap with agent + observability: dashboards and traces show that something went wrong, but they rarely close the loop to a verified fix. For the Google Cloud Rapid Agent Hackathon (Arize track), we wanted to treat Phoenix as more than storage—as something the agent can query at runtime—and pair that with a second agent that turns failures into concrete prompt and replay experiments.

That led to a reliability copilot mindset:

trace → understand → change → replay → judge

—with humans still deciding what ships.

What it does

TraceFix is a reliability copilot for Gemini (Google ADK) agents that uses Arize Phoenix as the system of record. When a task agent misbehaves or stalls, TraceFix can pull real traces via Phoenix MCP, reason about root cause, propose prompt and tool-routing improvements, replay the failing scenario with a candidate fix, and score the outcome (LLM-as-judge and/or structured checks).

The goal is a practical loop:

observe → diagnose → patch proposal → replay → verify → capture for regression

So observability is not only a dashboard, but an input to an improvement workflow. A subject agent (our store-ops / floor-terminal demo) generates realistic multi-step traces; the copilot focuses on turning those traces into actionable fixes humans can trust.

How we built it

We used Google ADK for both the subject agent and the debugger / copilot-style agent, with OpenInference instrumentation so LLM and tool spans land in Arize Phoenix.

The subject side is a single-store ops assistant: a small in-memory floor terminal simulator (search, inventory, cart, checkout, shipping and budget rules) so we get rich, reproducible failure modes without standing up a full e-commerce backend.

The copilot connects to Phoenix MCP (@arizeai/phoenix-mcp) to introspect traces at runtime, proposes changes, and uses a replay + LLM-as-judge path to compare before vs after behavior.

We also added a golden eval dataset (JSON scenarios + a runner) so “fixed” does not live only in a chat transcript—it can be re-run.

Challenges we ran into

Model variance vs deterministic simulators. The environment can enforce budgets and stock, but the LLM still has to click the right buttons; automated golden runs can flake depending on model and temperature, so we separated dataset as product artifact from CI-hard guarantees.
Wiring MCP cleanly. Getting the copilot to reliably use MCP tools alongside custom tools (replay, eval) required careful prompting and defensive handling when Phoenix or the MCP server is not running.
Scope control under hackathon time. We repeatedly chose a thin but end-to-end slice (real traces, real replay, real scoring signal) over a wide surface area (full hosted triage product), so judges could follow one story from failure to evidence to attempted fix.

Accomplishments that we're proud of

Closed-loop debugging, not “more charts.” We connected runtime agent behavior to Phoenix traces and made the copilot able to query that history through MCP, not just summarize from memory.
Replay + evaluation as first-class steps. After proposing a fix, we can re-run the scenario and compare outputs with a structured LLM-as-judge signal—so “better” is argued with evidence, not vibes.
A subject agent that’s intentionally “hard.” The in-store simulator includes inventory, budgets, shipping thresholds, and cart/checkout paths—so failures are interesting (wrong tool order, budget violations, stockouts) rather than toy chat failures.
Regression-ready artifacts. We captured scenarios in a golden eval dataset so improvements can be re-run and tracked over time, not lost in a single demo thread.
Hackathon-aligned architecture. Code-owned ADK agents + OpenInference instrumentation map cleanly to the Arize track’s emphasis on tracing, MCP, and evals.

What we learned

MCP is a UX for models. Treating Phoenix like a “database of runs” the copilot can query changes debugging from a human-only activity into a tool-using activity—if you design prompts and guardrails carefully.
Tracing pays off when tied to actions. Spans become valuable when they directly support RCA → replay → score; otherwise teams drown in spans without decisions.
Evals should match failure modes. One scalar score hides multi-dimensional failures; separating task success, tool correctness, and grounding makes iteration faster—even if models are noisy run-to-run.
Human approval is a feature. Auto-apply is exciting, but the winning pattern is human-in-the-loop promote: propose, prove on replay, then ship—especially when fixes touch customer-facing policy behavior.

What's next for TraceFix

Richer evaluators beyond a single judge score: explicit rubrics for tool order, policy adherence, grounding checks against tool outputs, and cost/token discipline.
Safer “promote” automation with snapshots: versioned policy/prompt overlays, one-click rollback, and an audit trail tied to Phoenix prompt versions and datasets.
Triage UI for non-engineers: failed runs, RCA summaries, replay buttons, diff view of prompt/routing changes, and promotion workflow—local first, then Cloud Run-style hosting.
Stronger regression suite fed from production-like traces: auto-curate failing traces into datasets, run nightly on CI, and surface drift before users do.
Deeper Phoenix workflows: experiments comparing prompt variants, cohort analysis by failure type, and clearer links between span evidence and patch rationale.