Talk2Build

A meeting agent for Alexa+: the team talks, and it turns their decisions into plans, designs, a spec and a tested prototype. Nothing that writes code runs until a human says yes.

Comment

Inspiration

Most meetings end with a to-do list that someone has to turn into work by hand. We wanted the opposite: an assistant that sits in the call, listens for real decisions, ignores the small talk, and starts the work while people are still talking. It shouldn't make anyone type a command, and it shouldn't do anything risky without asking. Voice-first Alexa+ plus MCP tools felt like the right shape for that.

What it does

Talk2Build is a meeting room with an agent panel. The demo case is a pizza-deadline game jam: Maya, Chris and Priya want a tiny browser game, Space Cat Courier, before the pizza arrives.

  • Says the goal, gets a plan. Maya states the goal out loud. A Plan card appears, and the decision and action items are saved under her name. Every requirement remembers who said it.
  • Designs land while people talk. When Chris describes the architecture and Priya asks for a wireframe, Talk2Build writes SVG artifacts. They render inline in the cards and appear on a shared stage in the meeting.
  • Live evidence, human decision. "Phaser or PixiJS, which is more popular?" produces an evidence card with live GitHub stars, open issues, last push and npm weekly downloads, each with its source URL. Talk2Build recommends; the humans decide on plain canvas with zero dependencies, and that decision is recorded.
  • Chatter gets parked. "Is the pizza here yet?" goes into a parked list and never enters the plan.
  • Feasibility check. "Check that it's feasible on a phone" produces a feasibility card.
  • A human-yes gate. Anything that writes code waits for a spoken "Yes". After Maya says yes, Talk2Build writes docs/spec.md from the meeting's decisions, scaffolds the prototype from its game-jam template, and runs the real node --test suite (7 of 7 pass). Progress shows on the card.
  • Playable in the meeting. The prototype runs in a sandboxed iframe on the shared stage (strict CSP, scripts only). In the video it is played: dodge asteroids, grab pizza for fuel, and every ten parcels the asteroids speed up.
  • Real speech, not only scripts. The mic feeds the same pipeline. With TALK2BUILD_STT=whisper, a clip is transcribed by a local faster-whisper worker (no API key) at POST /api/meetings/:id/transcribe. Without it, the browser's Web Speech API gives live captions. Each line keeps its speaker (from the Speaking-as picker) and is tagged with how it arrived. A recorded "Goal for this meeting…" clip, transcribed by Whisper, lands as Maya's turn and opens the Plan card, and spoken chatter is parked.
  • Minutes and files stay. Ending the meeting writes minutes. The spec, designs, prototype and test file stay in the meeting's workspace.

The UI is English by default, through a small zh/en dictionary (NEXT_PUBLIC_UI_LANG=zh switches to Chinese). The voice input still understands Chinese trigger phrases.

Honest scope. The video's meeting is scripted (the same script is behind the Play the pizza game jam button), and real-speech mode was added afterwards and is not shown in the video. The demo runs in mock mode (LLM_PROVIDER=mock, TALK2BUILD_AGENT=mock). The meeting lines are scripted, and agent output comes from deterministic fixtures, so no model is called. The Space Cat Courier game comes from a template written ahead of time: the meeting selects it and fills the spec, but it is not generated live. The video, the spec and the in-game footer all say so. The evidence numbers and the test run are real. The gameplay in the video is driven by a scripted keyboard bot.

How we built it

  • Alexa+ path: a self-hosted MCP server, protocol 2025-11-25, over Streamable HTTP at /mcp (POST/GET/DELETE, MCP-Session-Id, JSON or SSE responses). The browser uses a simulated Alexa+ client (src/alexa/client.ts) that initializes a session and calls the same tools over HTTP, which is the simulated-experience path the rules allow. alexa/addon-package/addon.json is the add-on manifest for a real Alexa+ client. We did not deploy it, because the add-on tools are partner preview only.
  • Tools: plan, design drafts (SVG), feasibility checks, research and evidence (live GitHub and npm APIs, cached for 10 minutes), scaffold_code and run_tests (spawns node --test), behind a confirmation gate.
  • Speech-to-text: a provider interface (src/lib/transcribe) with local Whisper (a persistent faster-whisper worker that decodes any format via ffmpeg) and the browser Web Speech API. Amazon Transcribe streaming is a reserved provider, not yet implemented.
  • Planner: a pluggable model layer, with three providers:
    • deterministic mock (the default, used in the demo);
    • Amazon Bedrock Converse through @aws-sdk/client-bedrock-runtime (LLM_PROVIDER=bedrock, default model amazon.nova-lite-v1:0);
    • an OpenAI-compatible endpoint. If a provider fails, the turn falls back to the mock and the client is told.
  • App: Next.js 15 / React 19 / TypeScript / Tailwind, with storage in SQLite via node:sqlite. Produced files are served raw with a sandbox CSP, nosniff and no-store.
  • Quality: Vitest suite with 164 passing tests (2 skipped), including an HTTP end-to-end run of the game-jam meeting. tsc and eslint are clean.
  • Video: recorded from the production build with Playwright, narrated with Kokoro TTS. Captions and loudness normalisation (-16 LUFS) are done in ffmpeg, and the narration was checked by transcribing it back with Whisper. The scripts are in demo/v2-pipeline/.

Challenges we ran into

  • Telling real work from chatter without making people speak in commands. A phrase like "we're going with…" first triggered a write confirmation that blocked the later "Yes".
  • Keeping the demo honest. An early script line claimed Phaser had more GitHub stars. The live card showed PixiJS ahead, so we rewrote the line.
  • Making a generated page playable inside the meeting without trusting it: a sandboxed iframe plus a strict CSP on the raw file route.
  • Alexa+ device deployment and AWS credits were unavailable to us (partner-only add-on tools; credit codes gone as of October 7), so the Alexa+ device path and live Bedrock calls could not be exercised.

Accomplishments that we're proud of

  • A meeting that goes from a spoken goal to a spec, two designs, a live evidence card and a tested, playable prototype in about two minutes, with a human-yes gate in front of anything that writes code.
  • Evidence that cites its sources and changes with reality instead of a canned chart.
  • A full English UI without losing the Chinese path, covered by tests.

What we learned

  • MCP over Streamable HTTP is a clean seam: a meeting agent maps naturally onto tools, and one endpoint serves both the simulator and a real Alexa+ client.
  • Confirmation is a product feature: people trust the agent more when the risky step visibly waits for them.
  • Honesty labels such as "demo mode" and "scaffolded from a template" make a demo more convincing, not less.

What's next

  • Run the Bedrock planner live (Nova Lite) instead of the mock, and measure latency against the Alexa+ budget.
  • Deploy the add-on to a real Alexa+ client once the tools are available.
  • Amazon Transcribe streaming as a speech provider (with AWS credentials), streaming chunks instead of whole clips, and speaker diarization instead of the manual Speaking-as picker.
  • Generating prototypes beyond the game-jam template.

Built With

Share this project:

Updates

Submission history