TracePilot

A Phoenix MCP operator loop that diagnoses a Gemini/ADK shopping agent from traces, fixes silent failures, and proves a 71→100 recovery.

Comment

Inspiration

Agent demos often look successful even when the trace shows the agent skipped the real requirement. TracePilot targets that quiet failure mode: not “did the model answer fluently?”, but “did the agent actually use tools and satisfy the task?”

What it does

TracePilot is a Phoenix MCP-powered operator loop for a Gemini / Google ADK shopping agent. The agent is organized as a coordinator with ADK sub-agents for product selection and purchase verification. TracePilot reads Phoenix/OpenInference proof artifacts, scores concrete completion checks, diagnoses failures, and turns the trace into a safer next run.

The hosted demo is a safe judge-testable replay. It does not expose live API keys, but it shows the verified proof sequence: initial 71/100, repeat 71/100, then 100/100 after the tool observations and prompt behavior were tightened.

How we built it

  • Gemini powers the shopping-agent reasoning path.
  • Google ADK defines the coordinator and sub-agent structure.
  • Phoenix/OpenInference captures agent, LLM, and tool spans.
  • Arize Phoenix MCP retrieves trace context for inspection.
  • A deterministic TracePilot operator scores the run and writes safe diagnosis artifacts.
  • Cloud Run serves the hosted proof shell.

Challenges

The hard part was keeping the proof honest. A polished final answer was not enough: the trace had to show search/click usage, product inspection, size-M preservation, and a clear explanation of tool steps. We also kept public surfaces sanitized so judges can test the story without exposing credentials.

Accomplishments

  • Public repo with Apache-2.0 license and runnable proof checks.
  • Hosted demo with a “Run safe demo proof” mode.
  • ADK coordinator plus product-selection and purchase-verification sub-agents.
  • Phoenix MCP proof narrative showing the recovery from 71/100 to 100/100.

What’s next

Next we would turn the proof loop into a broader evaluation harness across multiple tasks, add richer Phoenix-backed trend views, and connect the hosted UI to a live authenticated runner.

Built With

Share this project:

Updates