TraceCoach Agent

A Gemini agent that exports Phoenix traces, evaluates its own run, and turns weak spans into a self-improvement memo.

Comment

Inspiration

AI agents are becoming powerful enough to act, but they are still difficult to trust. When an agent gives a vague answer or makes a weak decision, teams often cannot see which step failed, which tool call mattered, or how to improve the next run.

TraceCoach Agent was inspired by this problem: production agents need an improvement loop, not just a chat interface.

What it does

TraceCoach is a live Gemini-powered agent that runs an operational task, exports OpenTelemetry traces to Arize Phoenix, evaluates its own behavior, and generates a self-improvement memo from the observed run.

The demo shows an agent handling a vague incident-response answer. It:

  1. Creates an action plan with Gemini.
  2. Executes a controlled workflow.
  3. Exports spans such as agent.plan, agent.execute_tools, phoenix.evaluate_trace, and phoenix.mcp_self_introspection.
  4. Scores the run for actionability, trace groundedness, goal fit, and tool efficiency.
  5. Produces a memo that turns weak spans into the next prompt or workflow experiment.

How we built it

The app is built as a Cloud Run web service with an Express backend and React frontend. Gemini handles live agent planning and execution. OpenTelemetry exports traces to Arize Phoenix using the Phoenix Cloud OTLP endpoint. The UI is designed for a judge-friendly demo: it shows the mission, live execution mode, evaluation scores, observed spans, and the self-improvement memo in one flow.

Challenges we ran into

The hardest part was making the demo both real and easy to understand in a short video. We had to make sure Gemini live mode worked, Phoenix tracing was enabled, Cloud Run secrets were configured correctly, and the UI clearly communicated the observe → evaluate → improve loop.

Another challenge was making the trace story visible without requiring judges to inspect backend logs. The final UI includes recording captions and a summary panel so the agent’s reliability loop is understandable at a glance.

Accomplishments that we're proud of

We are proud that TraceCoach is not just another chatbot demo. It shows a practical reliability pattern for agents: observe the run, evaluate the run, and improve the next run. The project is deployed publicly, uses live Gemini calls, and exports telemetry to Phoenix.

What we learned

We learned that observability is not only for debugging after something breaks. For agents, observability can become part of the product itself. Traces, evaluations, and self-review can help agents become safer, more transparent, and easier to improve.

What's next

Next, TraceCoach can be extended into a full agent reliability workbench. Future versions could query Phoenix traces through MCP directly, replay failed runs, compare prompt experiments, and automatically create regression tests from weak spans.

Built With

Share this project:

Updates