DemoMotion × WebMCP — Collaborative Demo Studio

## Inspiration

Product demo videos are painful to make, and AI agents are surprisingly bad at making them: an agent staring at a demo-generation UI has to guess which button does what. WebMCP flips that around — a website can expose its real actions as structured tools. We asked: what if a human and an agent could build a demo video together, in the same workspace, instead of the agent flailing at a UI or taking over completely?

## What it does

DemoMotion AI turns a web app URL and a goal into a narrated product demo video (it plans a scenario, records the browser with Playwright, writes and voices the narration, and renders an MP4 with ffmpeg).

For the WebMCP Challenge we added a shared /webmcp workspace where a human and a browser AI agent collaborate through five WebMCP tools:

get_demo_state — read the current demo (read-only)
create_demo — start an editable draft (title, steps, first-pass narration)
update_demo_narration — rewrite the narration/title; the UI updates instantly
generate_demo — run the real pipeline to a narrated MP4
export_demo — local-only download, gated on explicit human approval

The human can edit the exact same fields by hand, a "WebMCP enabled" badge and an agent/human activity log make every tool call visible, and every call actually executes against the same backend as the main dashboard — nothing is faked.

## How we built it

WebMCP is a thin adapter, not a rewrite. Tools register on document.modelContext (native in Chrome 149+ and the ChatGPT desktop browser, polyfilled elsewhere via @mcp-b/webmcp-polyfill) and call the existing FastAPI services rather than duplicating pipeline logic. A tiny shared store lets the React UI and the agent's tools mutate one source of truth, so agent actions are immediately visible and the human keeps control.

We added just enough backend surface for real collaboration: a lightweight POST /v1/demo-drafts that reuses the existing planner + narration writer (no recording), and optional title_override / narration_override on the job API so a narration edited before generation actually reaches the rendered video.

Frontend: Next.js 16, React 19, TypeScript
Backend: FastAPI, Playwright, ffmpeg, gTTS / Google Cloud TTS, optional Vertex AI (Gemini) for planning
Ops: Docker Compose, Terraform, Google Cloud Run

## What's new vs pre-existing

The generation pipeline, the dashboard, the demo-app, the REST API, and the deploy tooling existed before 2026-08-25. Everything under apps/web/app/webmcp and apps/web/lib/webmcp (the 5 tools, shared store, strict schemas), plus the /v1/demo-drafts endpoint and the override fields, is new for this challenge.

## Challenges we ran into

Making edits real. An agent rewriting narration is meaningless unless it changes the output. We threaded a narration override through the pipeline so the edited script is exactly what gets voiced and rendered.
Treating WebMCP as an external interface. Every tool uses a strict allow-list schema, returns structured errors (INVALID_INPUT, PROJECT_NOT_FOUND, GENERATION_FAILED, EXPORT_NOT_READY, EXPORT_REQUIRES_CONFIRMATION), accepts only http/https URLs, and never leaks secrets or stack traces.
Keeping the human in control. Export is a local download only and requires explicit confirmation — the agent can never publish or share on its own.
Runtime quirks. Tying tool lifecycle to an AbortController surfaced an expected AbortError under React StrictMode's dev double-mount; we handle it cleanly.

## What we learned

WebMCP shines when a site exposes creative actions with sharp boundaries. Shared state between human and agent — not full autonomy — is the interesting design space, and strict schemas plus human-gated side effects make an agent safe to hand real controls.

## What's next

More editable dimensions (scene ordering, pacing), and richer live progress in the workspace — while hosted projects, sharing, and team features stay in DemoMotion Cloud. WebMCP stays an open ecosystem feature.