VeraTest

• 

  Screenshot of result page

• 

  Structure of the synthetic Page Audit

• 

  Test Page

• 

  Test page

Inspiration

Every PM or Marketer has shipped a landing page on gut feeling. Real A/B tests need thousands of visitors and weeks of patience — most teams have neither, so design debates get settled by whoever argues loudest. Meanwhile, usability research has known since Nielsen (1994) that ~5 independent evaluators find most usability issues, and Condorcet's jury theorem says independent judges with better-than-random accuracy converge on the right answer as the panel grows. That gave us the bet behind VeraTest: a panel of 20 independent, persona-constrained AI evaluators should beat one clever prompt — and be auditable enough to trust.

What it does

Upload one or two landing-page screenshots plus a conversion goal. Six agent phases run: a Study Designer parses the goal and images into a brief, a Panel Recruiter synthesizes 3–7 audience personas and allocates 20 simulation agents across them, 20 Cognitive Walkers each evaluate the design as their persona (scoring six resonance dimensions, 1–10), a Bias Auditor checks the panel for confidence collapse and score inflation, an Insight Analyst clusters friction into themes, and three Report Narrators write the PM-ready output. A seventh agent — the FidelityAuditor — then runs LLM-as-a-Judge over every walker: did agent #13 actually reason like a "time-pressed founder", or did it drift into generic UX-expert voice? Every verdict lands as an EVALUATOR span and annotation in Arize Phoenix.

How we built it

• Stigmergy over frameworks. No LangGraph, no CrewAI. Agents coordinate like ants: each reads the shared Run document in SQLite and writes only its own slice. The whole pipeline is debuggable with a single SQL query.
• Gemini at every layer: gemini-2.5-flash for design/synthesis phases, gemini-2.5-flash-lite for the 20 parallel walkers and the judge — multimodal calls passing the actual screenshots.
• Agent Builder front door: an ADK LlmAgent ("VeraTest Concierge") exposes the pipeline as tools and mounts the @arizeai/phoenix-mcp server as a live MCP toolset, so the agent can answer questions about its own traces.
• One trace tree per run in Phoenix: run root → phase spans → 20 walker spans → every Gemini call, retry event, and judge verdict. ~139 spans per run, navigable in one click.
• Validation harness against real A/B tests with documented winners: when VeraTest commits to a verdict it's right 9/10 (90%) vs one-shot Gemini's 70% — small n, reported honestly with the full per-case matrix published.

Challenges we ran into

• Gemini free-tier 503 storms. Capacity spikes turned panels into silent abstentions. We fixed it three ways: tenacity retries with backoff recorded as span events (failures are visible in Phoenix, not hidden), a quorum gate — if completed sims $/$ panel size $< 0.7$ the run fails loudly rather than synthesizing a thin-panel verdict — and honest reporting (abstentions count against us in the headline accuracy number).
• Orphan traces. Early on, one run produced ~24 disconnected traces in Phoenix. Restructuring to a single AGENT-kind root span with phase children — including the background fidelity phase that outlives the pipeline — was fiddly async-context work, but it's now the best part of the demo.
• Cloud Run's sharp edges. CPU throttling silently killed OTLP export (fixed with --no-cpu-throttling), and multi-instance autoscaling split-brained our ephemeral SQLite — a run would complete on one instance and 404 on its sibling.
• Judging the judges. An LLM judge can hallucinate, so we triangulate it with a deterministic code-based eval: a 9/10 resonance score paired with "I felt confused and frustrated" is flagged incoherent — a rule that runs in microseconds and can't lie.

What we learned

That observability is the product, not an add-on: once every persona's reasoning, every retry, and every judge verdict was inspectable in Phoenix, debugging turned into reading. That honest metrics beat flattering ones — publishing 45% headline accuracy with the 90% decisive accuracy and the raw abstention cause taught us more than any cherry-picked number would. And that 20 small, constrained agents coordinating through shared state are easier to reason about than one large orchestrated graph.

Built With

• a2a-protocol-endpoints
• and-the-@arizeai/phoenix-mcp-server-mounted-as-a-live-mcp-toolset.-backend:-fastapi-+-sqlite-(wal)-as-the-shared-state-?-"pheromone-trail"
• cloud
• cloud-build
• datasets
• deployed
• llm-as-a-judge-evals
• logging
• opentelemetry/openinference-instrumentation
• plus-a-standalone-mcp-server.-frontend:-next.js-14-app-router-with-sse-live-?-updates.-cloud:-google-cloud-run-(frontend-+-backend)
• pydantic-schemas
• python-3.13-(backend)
• secret-manager
• typescript/react-(frontend).-ai:-gemini-2.5-flash-+-flash-lite-via-vertex-ai-and-the-google-genai-sdk;-google-cloud-agent-builder-(adk)-llmagent-front-?-door;-arize-phoenix-for-tracing