Inspiration

Footwear is one of the most-returned categories in commerce, and it is bought on keywords and shop-floor rules of thumb. The signal that actually decides fit — how you move — never reaches the checkout. And the checkout never learns why the shoes came back. We wanted the measurement, the recommendation and the money loop in one place — and we wanted it to work for agents, not just people.

What it does

Forma is a gait lab in the browser that shows its work:

  1. Measure — Film a short clip or go live. MediaPipe pose analysis runs entirely on-device: cadence, foot strike and overstride for runners; reach, hip position and precision for climbers. Video never leaves the browser.
  2. Brief — Movement cannot see size, budget or goal, so the agent confirms those in a one-sentence brief before any search begins.
  3. Recommend — A gear agent researches candidates against your metrics, live rider reports and a real product catalog (Channel3), then lands a three-pick showroom — each pick citing a measurement, with real product photography and a stock-verified size check.
  4. Transact — Pick a shoe and Forma opens a PayPal order in-window; consent stays with the shopper. The server captures and verifies COMPLETED before anything counts as paid, then attaches shipment tracking.
  5. Unwind — Fit fails? The same agent refunds the capture and reopens the fitting. The agent that fitted you also un-buys for you.
  6. Interoperate — A signed Fit Passport carries your movement profile so other agents can check a shoe against it over REST (/api/passport/<token>/fit) or MCP (check_fit). Agentic commerce in both directions: agents that buy for you, and agents that can sell to you.

Judge path (three clicks): vertiqal.vercel.app → The till → approve a real sandbox PayPal order and watch the verified capture land in the live transaction ledger — no camera or fitting needed. /lab is the merchant view: AG Grid joining PayPal transactions to recorded fit outcomes.

How we built it

  • On-device movement analysis — MediaPipe Pose on every frame, in-browser. Metrics lock against a saved baseline so re-filming shows real change.
  • Tool-gated agent — Qwen-72B (Featherless, Vercel AI Gateway fallback) runs a visible step loop — research, live stock, rider reports, athletes — each depth layer a gated tool call. Grok vision reads keyframes and sole wear; hero-card captions generate on-device.
  • PayPal Developer Platform — Orders v2 create/approve/capture with server-side verification; refunds on fit-failure; shipment tracking; Billing Plans for Forma Pro (server-verified ACTIVE, HMAC-signed tier cookie); transaction search feeding the /lab ledger; and @paypal/mcp as the second server in the two-MCP demo (pnpm agent-shop:mcp).
  • Channel3 — structured product catalog plus merchant offers merged into the agent's searchProducts, and exposed to external agents via the find_products MCP tool.
  • AG Grid powers the /lab merchant view; the APIMatic PayPal Context Plugin kept the coding agent SDK-accurate against Orders and Billing.
  • Honest verification — stock checks run through a stealth browser (Solari, UK egress) and return screenshot plus observed-URL receipts, labelled "template, not verified" where applicable.

Challenges we ran into

  • Consent vs autonomy — the agent can prepare an order but must never complete it. PayPal's approval window is the permission boundary, and a capture only counts after server-side verification.
  • Retail reality — several UK retailers bot-wall automated browsers outright (403s), so catalog offers from unverifiable merchants sort last and picks reconcile to a verifiable same-shoe page.
  • Honest stock claims — a screenshot is evidence, not proof. The UI says exactly that.
  • Sandbox quirks — PayPal sandbox occasionally stalls between approve and review; the capture still registers server-side and the ledger catches up once transaction search indexes it.

Accomplishments that we're proud of

  • A real end-to-end loop: browser pose measurement → LLM fitting → live catalog → verified PayPal checkout → refund on misfit — with no accounts, no database, and video that never leaves the device.
  • Two-directional agentic commerce: Forma's own agent buys; external agents consume the Fit Passport over REST or MCP and stage their own PayPal orders (scripts/agent-shop.mjs).
  • PayPal is the protagonist, not a plugin: orders, refunds, shipment tracking, subscriptions, transaction insights and the Agent Toolkit are all exercised inside one product.

What we learned

Agents do not need checkout replaced — they need checkout with a consent boundary they cannot cross. PayPal's approval window plus server-side capture verification is exactly that boundary. The interesting product surface is everything around it: measurement, verification receipts, and the unwind path.

What's next for Forma

Longitudinal movement profiles (multi-baseline history and trends) as the Pro tier; richer dispute handling through the Agent Toolkit; and publishing the Fit Passport MCP to agent directories so any third-party shopping agent can fit-check a shoe against a real measured body.

Built With

  • ag-grid
  • agentic-commerce
  • ai
  • apimatic
  • channel3
  • featherless
  • mcp
  • mediapipe
  • next.js
  • paypal
  • qwen
  • tavily
  • typescript
  • vercel
Share this project:

Updates

Submission history