Self-Improving Market Research Workflow

A self-improving multi-agent market-research system where every agent is traced, scored, versioned, and rolled back like code, governed end-to-end by Arize Phoenix.

Comment

Track: Arize · Event: Google Cloud Rapid Agent Hackathon Hosted app: https://surfcoders-244452546912.us-central1.run.app

Elevator pitch

A self-improving, fully-observable multi-agent system for product-launch market research; where the real contribution isn't the research, it's the governance layer that versions, evaluates, and improves every agent using Arize Phoenix as the backbone.

Inspiration

Multi-agent systems are easy to demo and hard to operate. The moment a prompt changes, you have no idea which agent regressed, why, or how to roll it back. We wanted to treat agents the way we treat code and containers: every agent has a version history, every change is evaluated, every regression is caught, and any version can be rolled back. They are all driven by real production traces, not guesswork.

Market research is the task. The governance plane that sits on top of it is the product.

What it does

SurfCoders runs a 5-stage market-research pipeline (frame → plan & retrieve → synthesize → challenge → reconcile), then governs it with a second loop built entirely on Arize Phoenix:

  • Execution plane (inner loop): worker agents plan, ground their answers with live Google Search, synthesize a market brief, adversarially challenge it (Skeptic), and reconcile the result. A ControlShellAgent routes between stages and retries/escalates on failure.
  • Governance plane (outer loop): an LLM-as-judge (GenericEvaluatorAgent) scores each stage; a PhoenixInspectorAgent reads the pipeline's own failed spans through the Phoenix MCP server and proposes targeted prompt fixes; and Agent Pages record the whole history.

Agent Pages is the differentiator. Each agent gets:

  • a full iteration timeline (color-coded by source: base / auto / developer / inspector),
  • an Image Registry: promoting an iteration mints an immutable, versioned prompt+score artifact (Docker-registry-style history),
  • one-click rollback to any prior image (non-destructive, creates a new iteration),
  • dependency flags: promoting an upstream agent auto-triggers regression checks downstream,
  • human-in-the-loop injection: developers add rules directly, separate from the budget-gated auto-improve cycle.

All of this runs as one FastAPI app: a brief-submission form with live streaming logs, and the Agent Pages dashboard at /agent-pages, sharing one process and one database.

How we built it

The entire stack runs on Google Cloud AI tooling: every agent is a Gemini 3 model served through Vertex AI, the app is deployed on Google Cloud Run, and Google's grounding tools feed live evidence into the pipeline.

  • Google Cloud AI stack: all agents call Gemini 3 on Vertex AI - gemini-3-flash-preview for retrieval/parallel work, gemini-3-pro-preview for synthesis/eval. Worker agents use Vertex AI's Google Search grounding tool to fetch real, citable market evidence. A single MODEL_TIER env var swaps tiers with no code changes; GOOGLE_CLOUD_LOCATION=global pins the Gemini 3 endpoint. The whole service is containerized and shipped to Google Cloud Run (--no-cpu-throttling for the long-running pipeline, single instance for in-memory run state).
  • Data sources: live Google Search grounding (via Vertex AI) for real-time market evidence, plus three curated golden datasets (llm_launch, physical_launch, edge_cases) registered in Phoenix and used to drive baseline-vs-improved experiments.
  • Observability (Arize Phoenix Cloud): every span emitted via OTLP with OpenInference semantic conventions; eval scores annotated on each stage span; prompts registered in Phoenix Prompt Hub and fetched at runtime (with local fallback); golden datasets and an experiment runner that records baseline-vs-improved deltas.
  • Self-introspection (Partner MCP): PhoenixInspectorAgent calls the @arizeai/phoenix-mcp server over MCP. Each call emits a TOOL span, so the agent reading its own telemetry is itself traced.
  • App & UI: FastAPI + server-rendered HTML/JS: a run form with background-thread execution + live-log streaming, and a native Agent Pages dashboard (timeline, registry, run history, inspector/inject controls) backed by a SQLite page store.

Challenges we ran into

  • Gemini 3 subtleties. It only serves from GOOGLE_CLOUD_LOCATION=global (404s in regional endpoints), leaves open-keyed dict schema fields empty (we switched those to lists), and occasionally returns empty text. We added a retry guard so structured-output calls never crash a downstream json.loads.
  • Phoenix Cloud behind a WAF redirected our MCP subprocess's default User-Agent; we patched the Node fetch UA so the inspector's MCP calls get through.
  • Cloud Run + a multi-minute background pipeline needed --no-cpu-throttling and careful streaming so the UI shows live progress instead of hanging on a synchronous request.
  • Keeping the final gate honest: tuning the evaluator/reconciler so the quality gate passes on real evidence rather than rubber-stamping.

Accomplishments that we're proud of

  • A generic, reusable governance architecture: retargeting it to a new task means changing eval_criteria, not rewriting agents.
  • Agent Pages: per-agent versioning, an image registry, rollback, and cascade-regression detection: capabilities most agent stacks simply don't have.
  • A genuine self-improvement loop where the agent reads its own production traces via MCP and proposes fixes that are recorded as tracked, scored iterations.
  • One deployable app unifying execution, live observability, and governance.

What we learned

  • Treating agents as versioned artifacts (promote / tag / roll back) makes a multi-agent system operable in a way that ad-hoc prompt editing never does.
  • Tracing is the substrate for self-improvement: once every stage is a span with eval scores, an agent can reason about its own failures.
  • Preview models trade capability for stability. Production agent systems need retry/empty-response guards and region-aware config as first-class concerns.

What's next for Self-Improving Market Research Workflow

  • Expand the golden datasets (5+ per scenario) for stronger experiment signal.
  • Shared/managed storage so the governance DB survives multi-instance scaling.
  • Bound tail latency on the reasoning agents (request timeouts + retry on stalls).

Built with

Google Cloud AI: Gemini 3 · Vertex AI · Vertex AI Google Search grounding · Google Cloud Run Arize: Phoenix Cloud · Phoenix MCP (@arizeai/phoenix-mcp) · Phoenix Prompt Hub · OpenInference App: FastAPI · OpenTelemetry · Python · uv · SQLite

Built With

  • arize
  • fastapi
  • gemini-3
  • google-ai
  • google-cloud
  • google-cloud-run
  • openinference
  • opentelemetry
  • phoenix-cloud
  • phoenix-mcp
  • python
  • sqlite
  • uv
  • vertex-ai
Share this project:

Updates