FormForge AI

⏱ Note for reviewers: a live upload runs the
▎ full pipeline on Cloud Run (cold start +
▎ Gemini multi-agent debate) and takes ~2-3
▎ minutes, with progress streaming live. For an ▎ instant, no-wait walkthrough, append ?demo=1

## Inspiration Most AI form-check apps give you a single, confident verdict — and confidence is exactly the problem. Real coaching is a negotiation between encouragement and rigor: a good PT keeps you motivated, while a sharp physiologist won't let a dangerous habit slide. We wanted that tension to be visible, not flattened into one bland answer.

And we wanted to prove a harder claim: that an agent can genuinely learn an individual user's coaching style — not just say it does, but show it with numbers.

## What it does You upload a short workout video. FormForge runs a multi-agent debate:

• The Encourager (warm certified PT, 10 years) finds what you did well and proposes one next step.
• The Scrutinizer (rigorous exercise physiologist, PhD) hunts for injury risk and biomechanical flaws, grounded in measured joint angles and tempo.
• The two debate in real time — agreeing, pushing back, standing their ground.
• The Mediator (Head Coach) synthesizes both sides with your past training context, retrieved at runtime through the Arize Phoenix MCP server, into a balanced
  verdict with prioritized action items.

Then you react (too harsh, too soft, perfect, or free text). An LLM-as-a-Judge evaluates the debate and the personas evolve bidirectionally. Over time, the two coaches become your personal critic pair.

## How we built it Built with Google ADK (the official open-source framework of Google Cloud Agent Builder) and deployed on Cloud Run (an Agent Builder-supported runtime), powered
by Gemini 2.5 Pro / 2.5 Flash and Gemini 3.5 Flash, with the Arize Phoenix MCP partner server for agent self-introspection.

• Orchestration: ADK hierarchical multi-agent (parallel debate rounds + convergence detection + mediator synthesis).
• Multi-modal × multi-agent: one video, multiple lenses — a 2-stage PoseExtractor uses MediaPipe to measure joint angles/tempo and Gemini 2.5 Pro to
  interpret them, so the LLM never fabricates precise numbers.
• Self-introspection (MCP): a custom FastMCP server wraps the Phoenix REST API + Firestore; the Mediator autonomously calls query_past_debates and query_similar_safety_flags to pull the user's own history.
• Observability: Arize Phoenix Cloud auto-instruments every agent — the full debate, the convergence judge, and the MCP tool calls all show up as a traced tree.
  
• Self-improvement, measured: a Phoenix Datasets & Experiments pipeline runs the same debate scenarios under the baseline persona (v1) vs. the evolved persona
  (v3) and scores them with two evaluators.

## Accomplishments that we're proud of We didn't claim personalization — we measured it. Using Phoenix Experiments on a fixed dataset, holding everything constant and changing only the persona version
(after 3 rounds of user feedback):

• Preference alignment (matches this user's style): 0.62 → 0.795 — +28% ↑
• Debate quality (coverage / evidence / actionability): 0.75 → 0.765 — held ✓

The coaching became yours — without degrading its substance. The whole comparison lives in Phoenix, timestamped and reproducible. We also shipped a genuinely
adversarial agent pair, runtime trace introspection via MCP, and a full video→pose→debate→consensus pipeline verified end-to-end on Cloud Run.

## Challenges we ran into

• A package-shadowing trap: our project mcp/ folder silently shadowed the PyPI mcp package, quietly killing the Mediator's MCP introspection. Fixed with import-path hardening.
• Bugs that only appeared on Cloud Run: missing GLES libs for MediaPipe, MCP stdout pollution corrupting the JSON-RPC channel, and a Phoenix 401 (the auth header needed Bearer, and the collector endpoint needed the space path). Each was invisible locally.
• Avoiding false precision: letting an LLM "eyeball" joint angles produces confident nonsense, so we split measurement (MediaPipe) from interpretation (Gemini).
  

## What we learned Observability isn't a dashboard you bolt on at the end — it's how you debug agents at all. Phoenix traces were how we found the silently-dying MCP path. And evals turn "it feels better" into "it's 28% better, here's the experiment" — the difference between a claim and a result.

## What's next

• Bad-form detection across more lifts (deadlift, bench, overhead press) with larger labeled eval datasets.
• Online evals + continuous persona tuning from live feedback.
• Reference architecture: the adversarial-pair + self-improvement loop generalizes well beyond fitness.

## Medical Disclaimer This tool provides informational analysis only. It is not medical advice. If you experience pain, injury, or persistent discomfort during exercise, consult a
qualified healthcare or fitness professional before continuing.

Built With

• arize-phoenix
• cloud-storage
• fastmcp
• firestore
• gemini-2.5-flash
• gemini-2.5-pro
• gemini-3.5-flash
• google-adk
• google-cloud-run
• mediapipe
• model-context-protocol
• python
• streamlit
• vertex-ai
• vertex-ai-vector-search