Bandmate-WebMCP

Inspiration

Four bars of melody and no idea what goes under it. Asking ChatGPT yields MIDI text to convert, import and audition five minutes later, and the model never heard your take. WebMCP suggested a third thing: the agent sits in the tab with the sequencer, reads your notes as structure, and writes into the bar you are still hearing.

What it does

Bandmate is a four-track loop sequencer (drums, bass, chords, lead; 4 bars, 64 steps) exposing 16 WebMCP tools. Play a riff on the piano or your computer keys; ask "what goes under this?"; the agent calls suggest_chords then add_chords, and teal notes cascade into the grid while the loop keeps playing. It adds bass and drums in three styles, and can play you two arrangements before committing one. Record live and its writers are unregistered until you stop. Export a Standard MIDI File into any DAW.

Why WebMCP fits this use case

• Shared state beats generated text. The agent needs what you just played, what you just muted, and which bar you are looking at. All three live in the page's store; get_session serialises them in under 700 characters.
• The human must stay in the loop. Harmony is taste. audition is a tool whose execute() awaits a click; the agent's turn pauses until a human has listened.
• Capabilities should track the mode. While you record, writers are dangerous; during an audition they would corrupt the preview. Tool groups are disposed via AbortController and the browser fires toolchange, so the agent's tool list is truthful at every moment.
• No backend. One static site works in ChatGPT desktop, Chrome with the flag, or with a bring-your-own-key agent.

How it improves the experience for people

Five minutes per iteration becomes eight seconds. You never leave the grid or convert a file; every agent write is teal beside your amber notes and undoable in one click. The agent reacts to what you did by hand ("muted bass 5s ago"), and when it cannot decide it plays A and B instead of guessing. Students see why a chord works: numerals, names, borrowed and altered chords, a why per suggestion.

What humans and agents can accomplish together that was previously impossible

The agent hears your playing as structure, not audio, so it can answer "what goes under what I just played?" and drop the answer into the bar you are still hearing. A DAW has no ear for theory; a music generator has no room for you. And because the agent cannot hear, it asks your ears: it plays two options in your tab and waits for you to pick.

How I built it

Vite + vanilla TypeScript, Tone.js 15, no server.

• Registration. Each tool is a typed defineTool({name, description, inputSchema, annotations, execute}) registered via document.modelContext.registerTool(descriptor, {signal}). That is the only surface ChatGPT's browser and Chrome 152 expose; navigator.modelContext is probed only as a fallback for other builds.
• Tool groups and toolchange. Five ToolGroups (readers, writers, dialog, transport, recording), each with one AbortController. Recording or auditioning disposes the writers; leaving re-registers them. Abort unregisters natively, the browser emits toolchange, and the header pill shows idle 15 / recording 8 / auditioning 7.
• Annotations. Readers carry readOnlyHint. Exactly one tool, clear_track, carries consequentialHint: it is the only whole-track wipe, and it also asks the human first. untrustedContentHint appears nowhere, because no Bandmate content originates from a third party.
• Schema hygiene. Every schema is type:'object', additionalProperties:false, explicit required, enums for closed sets, numeric bounds, a description on every property. An Ajv strict test compiles all of them and enforces name ≤ 30, description ≤ 350, parameter ≤ 120.
• Bounded results. get_session ≤ 700 chars, get_track ≤ 1,500, everything else clipped with truncated:true. Notes use a compact b1: 1@1 3@5 5@9 degree format; drum lanes are 16-char velocity-class strings. Every result ends with hint or next.
• Audition as an awaiting tool. Each option is a full variant Session swapped into the store's transient layer at the next step; the engine tick reads store.current(), so previews are audible without rebuilds. Keep commits one undoable agent mutation; the promise resolves with chosen, listened, undo_id. On timeout it returns awaiting_human and the card stays open. Honours AbortSignal.
• Attribution and undo. Note.by / DrumHit.by are set by store.apply() from the caller's meta. Undo is a linear stack of deep-cloned snapshots (cap 50) tagged with author and label; the undo tool refuses when the top entry is human. Writers that would remove human notes call store.dryRun() then askHuman().
• One clock. A single 64-step Tone.Sequence reads store.current() every 16th; no Parts, no rebuilds. Recording quantises keydown times to the nearest 16th and writes by:'human' live.
• Export. Pure SMF type-1 writer (PPQ 480, swing baked, drums on channel 10). The tool sets the Download button's href and spotlights it; the click is the human's.

Challenges

document.modelContext hands back inputSchema as a JSON string and executeTool takes one, which shaped the built-in agent and the Playwright harness. Keeping a promise pending inside execute() for up to 60 s and defining what happens after a host timeout (card stays; late click applies). Making agent writes audible on the next 16th without glitches. Autoplay: an agent transport play before any gesture must return muted_until_user_taps and spotlight Play.

Accomplishments

Sixteen tools that plain sentences route to across 12 eval cases; an audition whose promise waits for a human ear; a mode-driven tool list the browser reports truthfully; zero backend.

What I learned

Tool descriptions are UX: verb-first, one "use when", one constraint, and "returns undo_id" changed which tool the model picked more than schema tweaks did. Hinted results turn three calls into a conversation. Once every cell knows who wrote it, undo, confirmation and "what did I just do?" fall out for free.

What's next

WebMIDI input, more styles and time signatures, a per-bar "why this chord" explainer, auditions of fills and voicings, all as more tools on the same store.

Built with

webmcp typescript vite tone.js web-audio midi vitest playwright github-pages chatgpt-desktop

Testing instructions for judges

1. Open the live URL in ChatGPT desktop (macOS, in-app browser) or Chrome 149+ with chrome://flags/#enable-webmcp-testing. Other browsers: open the agent drawer, paste your own OpenAI or Anthropic key, use the built-in agent.
2. Press ▶ Play once (audio unlock). The demo riff loops.
3. Ask Add lo-fi drums → teal drum cells. What goes under what I just played? → chord strip fills. Let me hear two bass options → press ▶ A, ▶ B, Keep one.
4. Press ● Record: the pill drops 15 → 8 tools. Play keys (a w s e d f …), press Stop; back to 15.
5. Ask Export this as MIDI, click the highlighted Download button, open the file in any DAW.
6. Chrome: DevTools → Application → WebMCP lists the registered tools with annotations (15 idle, 8 recording); ?debug=1 opens the inspector. Repo: pnpm test.

Tools reference: docs/tools.md, generated from the schemas.

Built With

• astra
• chatgptdesktop
• codex
• github
• midi.js
• playwright
• tone.js
• typescript
• vite
• vitest
• web-audio
• webmcp