Inspiration
We built an AI video pipeline that generates continuity-preserving scene sequences and scores each scene with a five-expert review panel (Director / DP / Colorist / Continuity / Editor). It worked — but once a project grew past a few dozen cuts, a real problem emerged: the scores scattered across JSON files with zero observability. When average quality dropped, nobody could answer the only question that mattered — which scene regressed, and why? Continuity was judged subjectively, with nothing objective to check it against.
That is precisely the problem Arize Phoenix exists to solve for LLM systems. Our evaluation pipeline is an LLM evaluation pipeline — five LLM judges scoring an artifact across iterations. The mapping wasn't a stretch; it was the missing half of the system.
What it does
The agent turns scattered evaluations into an observable, diagnosable loop:
- Ingest — every scene evaluation is logged to Phoenix as an OpenInference trace: one CHAIN span per scene, five LLM child spans (one per expert), plus per-expert score annotations. Crucially, continuity is also measured, not just judged — deterministic metrics (subject_consistency via dHash, palette_stability via RGB histogram) are logged on the same span as a separate CODE annotation.
- Diagnose — QualityAnalyst (Gemini, via Google ADK) connects to Phoenix through the @arizeai/phoenix-mcp server and autonomously calls MCP tools (list-projects, list-traces, get-spans, get-span-annotations) to find regression patterns — e.g. "scene 03 stalls at 6/10 on character_consistency across 3 iterations." It then cross-checks the subjective expert scores against the deterministic continuity metrics on the same span: measured drift confirms a real defect; a gap flags judging variance instead.
- Prioritize — the agent proposes which scene to fix first and what to change.
It doesn't just chat — it calls partner MCP tools and reasons over real trace data.
How we built it
- Google ADK (google-adk 2.1.0) is the agent framework. McpToolset + StdioConnectionParams attach the Phoenix MCP server as a live toolset — verified to load 27 real MCP tools over a genuine stdio handshake (not lazy stubs).
- Gemini is the agent's reasoning backbone (free AI Studio GenAI tier — no Vertex billing).
- Arize Phoenix (self-hosted, free) is the observability layer. phoenix_ingest.py writes evaluations as OpenInference spans + annotations over OTLP.
- OpenInference semantic conventions give the spans their CHAIN / LLM / CODE structure so Phoenix renders them as proper expert panels.
- A deterministic continuity_metrics.py (dHash + RGB histogram, zero external deps) provides the objective ground-truth the agent cross-checks against.
We isolated two virtualenvs because Phoenix pulls a newer OpenTelemetry than ADK allows — the two connect over MCP/stdio, so they never need to share a process.
Challenges we ran into
- OpenTelemetry version conflict — arize-phoenix bumps opentelemetry past what google-adk pins. Solved by splitting into two venvs joined over MCP rather than forcing one environment.
- ADK 2.x API drift — the current API is McpToolset (not the deprecated MCPToolset) with StdioConnectionParams(server_params=...); older docs were wrong. We confirmed against the live package.
- Making continuity objective — "the character looks different" is a vibe until you measure it. We added dHash subject consistency + RGB-histogram palette stability so the agent has something real to cross-check expert opinion against.
- Mapping a film-review panel onto observability primitives — getting scene → trace, expert → span, score → annotation to feel native in the Phoenix UI took iterating on the span schema.
Accomplishments that we're proud of
- End-to-end works on real data: 27 traces / 162 spans ingested, with expert_score aggregating to μ 8.71 in the Phoenix Stats panel — the per-scene, per-iteration expert structure renders exactly as designed.
- The agent makes unprompted MCP tool calls — list-projects → list-traces — then correctly diagnoses a stalled scene without being told where to look.
- Continuity is measured, not guessed — a CODE annotation sits beside the five expert opinions, turning subjective review into something verifiable.
- Fully free to run — Phoenix self-hosts, Gemini uses the free tier; git clone + one script reproduces the whole stack.
What we learned
Observability isn't a dashboard you bolt on at the end — it's what makes an evaluation loop closeable. Once every score is a queryable span, an agent can do the triage a human can't at scale. And the most valuable signal isn't the LLM judges alone — it's a cheap deterministic metric sitting next to them so the agent can tell a real regression from judging noise.
What's next for AI Video Quality Agent
- Promote ingest to a continuous loop so the agent monitors live pipeline runs, not batch snapshots.
- Use Phoenix Experiments to A/B storyboard-prompt revisions and let the agent pick winners.
- Close the loop: feed the agent's fix recommendations back into the storyboard compiler for automatic re-rendering.
- Multi-project fleet view across many video series.