Trace Forge

Inspiration

React code reviews are tedious and inconsistent. Junior devs get inconsistent feedback; senior devs waste hours on mechanical checks. We wanted an agent that doesn't just review code — it gets better at reviewing code the more it runs, closing the loop between execution and improvement automatically.

What It Does

TraceForge reviews React components across 4 dimensions — performance, accessibility, best practices, and security — then evaluates its own output and updates its review strategy after every run. Each review is traced end-to-end in Arize Phoenix Cloud so you can inspect every LLM call, tool invocation, and token count.

The self-improvement loop:

Review → agent analyzes component across 4 dimensions
Evaluate → LLM-as-Judge scores completeness, accuracy, actionability, calibration
Reflect → agent queries its own past traces via Phoenix MCP
Update → agent writes strategy adjustments to Firestore (v1 → v2 → v3 ...)
Repeat → next review loads latest strategy → improved coverage

After 50+ reviews the agent has updated its strategy to v58, adding specific checks for icon-only button labels, memoization chain verification, inline style detection, and dozens more — none of which were in the original prompt.

How We Built It

Agent runtime: Google ADK (Python) with LlmAgent orchestrating all tool calls
LLM: Gemini 3 Flash Preview via Vertex AI (location=global — required for Gemini 3)
Tracing: openinference-instrumentation-google-adk → Arize Phoenix Cloud for full observability
Self-introspection: @arizeai/phoenix-mcp — the agent queries its own past traces to reflect on past performance
Web framework: FastAPI + Server-Sent Events for real-time streaming
Frontend: Next.js 15 App Router + Tailwind CSS v4 + Recharts for eval trend charts
Persistence: Cloud Firestore (reviews, strategy versions, eval history)
Hosting: Cloud Run (two services — agent backend + Next.js frontend)

Challenges We Ran Into

Gemini 3 availability: Gemini 3 models are only available via location=global on Vertex AI — standard regional endpoints return 404. Took some digging to discover this.
Sync Firestore in async FastAPI: Firebase Admin SDK is synchronous only. We used run_in_executor to call Firestore without blocking the async event loop.
SSE through Cloud Run: Getting Server-Sent Events to stream correctly through Cloud Run's HTTP/2 required careful tuning of buffer and flush settings.
LLM-as-Judge reliability: Prompt-engineering the eval rubric so scores were consistent and actionable — not just a number, but a signal the agent could act on.

Accomplishments That We're Proud Of

A fully autonomous self-improvement loop that genuinely works: strategy v1 → v58 with measurable gains in eval scores across 50+ runs.
End-to-end observability via Arize Phoenix — every token, every tool call, every eval score is inspectable.
Live streaming review UI that shows the agent's reasoning in real time via SSE.

What We Learned

Google ADK is remarkably expressive for multi-tool agents; the LlmAgent abstraction handles tool routing cleanly.
LLM-as-Judge is a powerful eval primitive when the rubric is carefully designed — subjective dimensions like "actionability" and "calibration" are hard to measure any other way.
Arize Phoenix's MCP server is a game-changer: letting the agent introspect its own traces closes the feedback loop without any human in the loop.

What's Next for TraceForge

Expand beyond React to TypeScript/Node and Python codebases
Add multi-model comparison (run the same component through Gemini and GPT-4o, compare evals)
Let users subscribe to strategy updates — get notified when the agent discovers a new class of bug
Open-source the self-improvement framework as a reusable ADK pattern