Inspiration
Footwear is one of the most-returned categories in commerce, and it is bought on keywords and shop-floor rules of thumb. The signal that actually decides fit — how you move — never reaches the checkout. And the checkout never learns why the shoes came back. We wanted the measurement, the recommendation and the money loop in one place — and we wanted it to work for agents, not just people.
What it does
Forma is a gait lab in the browser that shows its work:
- Measure — Film a short clip or go live. MediaPipe pose analysis runs entirely on-device: cadence, foot strike and overstride for runners; reach, hip position and precision for climbers. Video never leaves the browser.
- Brief — Movement cannot see size, budget or goal, so the agent confirms those in a one-sentence brief before any search begins.
- Recommend — A gear agent researches candidates against your metrics, live rider reports and a real product catalog (Channel3), then lands a three-pick showroom — each pick citing a measurement, with real product photography and a stock-verified size check.
- Transact — Pick a shoe and Forma opens a PayPal order in-window; consent stays with the shopper. The server captures and verifies COMPLETED before anything counts as paid, then attaches shipment tracking.
- Unwind — Fit fails? The same agent refunds the capture and reopens the fitting. The agent that fitted you also un-buys for you.
- Interoperate — A signed Fit Passport carries your movement profile so other agents can check a shoe against it over REST (
/api/passport/<token>/fit) or MCP (check_fit). Agentic commerce in both directions: agents that buy for you, and agents that can sell to you.
Judge path (three clicks): vertiqal.vercel.app → The till → approve a real sandbox PayPal order and watch the verified capture land in the live transaction ledger — no camera or fitting needed. /lab is the merchant view: AG Grid joining PayPal transactions to recorded fit outcomes.
How we built it
- On-device movement analysis — MediaPipe Pose on every frame, in-browser. Metrics lock against a saved baseline so re-filming shows real change.
- Tool-gated agent — Qwen-72B (Featherless, Vercel AI Gateway fallback) runs a visible step loop — research, live stock, rider reports, athletes — each depth layer a gated tool call. Grok vision reads keyframes and sole wear; hero-card captions generate on-device.
- PayPal Developer Platform — Orders v2 create/approve/capture with server-side verification; refunds on fit-failure; shipment tracking; Billing Plans for Forma Pro (server-verified ACTIVE, HMAC-signed tier cookie); transaction search feeding the
/labledger; and@paypal/mcpas the second server in the two-MCP demo (pnpm agent-shop:mcp). - Channel3 — structured product catalog plus merchant offers merged into the agent's
searchProducts, and exposed to external agents via thefind_productsMCP tool. - AG Grid powers the
/labmerchant view; the APIMatic PayPal Context Plugin kept the coding agent SDK-accurate against Orders and Billing. - Honest verification — stock checks run through a stealth browser (Solari, UK egress) and return screenshot plus observed-URL receipts, labelled "template, not verified" where applicable.
Challenges we ran into
- Consent vs autonomy — the agent can prepare an order but must never complete it. PayPal's approval window is the permission boundary, and a capture only counts after server-side verification.
- Retail reality — several UK retailers bot-wall automated browsers outright (403s), so catalog offers from unverifiable merchants sort last and picks reconcile to a verifiable same-shoe page.
- Honest stock claims — a screenshot is evidence, not proof. The UI says exactly that.
- Sandbox quirks — PayPal sandbox occasionally stalls between approve and review; the capture still registers server-side and the ledger catches up once transaction search indexes it.
Accomplishments that we're proud of
- A real end-to-end loop: browser pose measurement → LLM fitting → live catalog → verified PayPal checkout → refund on misfit — with no accounts, no database, and video that never leaves the device.
- Two-directional agentic commerce: Forma's own agent buys; external agents consume the Fit Passport over REST or MCP and stage their own PayPal orders (
scripts/agent-shop.mjs). - PayPal is the protagonist, not a plugin: orders, refunds, shipment tracking, subscriptions, transaction insights and the Agent Toolkit are all exercised inside one product.
What we learned
Agents do not need checkout replaced — they need checkout with a consent boundary they cannot cross. PayPal's approval window plus server-side capture verification is exactly that boundary. The interesting product surface is everything around it: measurement, verification receipts, and the unwind path.
What's next for Forma
Longitudinal movement profiles (multi-baseline history and trends) as the Pro tier; richer dispute handling through the Agent Toolkit; and publishing the Fit Passport MCP to agent directories so any third-party shopping agent can fit-check a shoe against a real measured body.
Built With
- ag-grid
- agentic-commerce
- ai
- apimatic
- channel3
- featherless
- mcp
- mediapipe
- next.js
- paypal
- qwen
- tavily
- typescript
- vercel