Duet

• 

Duet lets people sketch music while an AI agent reads the live session, adds deterministic harmony or bass, and explains every reversible change.

Inspiration

Music software can be powerful but intimidating: theory vocabulary, dense interfaces, and opaque “AI generate” buttons often put the person outside the creative loop.

I wanted to explore a different interaction: what if an AI agent could work inside a music editor, see the same live project as the person, and make small musical changes that stay understandable and reversible?

That became Duet.

What it does

Duet is a local-first, browser-based piano-roll editor for collaborative human–AI composition.

A person can draw notes, play from a MIDI keyboard or computer keyboard, set chords, select bars, and hear the result. In a compatible browser, ChatGPT/Codex discovers Duet’s WebMCP site tools directly from the page.

The agent can then:

• Read a concise summary of the live project or the exact selected notes.
• Analyze harmony deterministically.
• Set a chord progression with validated voicings.
• Add precise notes or generate bass, counter-melody, and pad material.
• Quantize, humanize, transpose, or remap a selected phrase.
• Change tempo and instruments.
• Create auditionable drafts and alternatives instead of silently overwriting music.
• Verify the resulting state after every action.

Every persistent AI edit appears in the app with a plain-language explanation and a one-click Undo action.

How I built it

Duet is a client-only React + TypeScript app powered by Vite and Zustand. The editor uses Pointer Events so drawing, dragging, stretching, and erasing notes work across mouse, touch, and pen input.

Musical rules are deterministic rather than model-generated. I use Tonal for chord and scale logic, Tone.js for browser playback, and @tonejs/midi for MIDI export. Optional Web MIDI support allows class-compliant keyboards to control the editor.

For WebMCP, Duet registers narrow site tools directly through document.modelContext.registerTool. The tools read the current browser state, validate every mutation, and create serializable inverse patches. That makes changes inspectable, reproducible, and undoable without a backend, API key, account, or cloud project storage.

Browser and MIDI constraints

A practical constraint shaped the product flow: ChatGPT/Codex’s built-in browser can discover and call WebMCP site tools, but it does not expose a connected physical MIDI device to the page in the same way a regular Chrome session can.

I designed Duet around a shared live-session model to bridge that gap. The musician can open Duet in a WebMCP-enabled Chrome session, connect and play a MIDI keyboard there, while ChatGPT/Codex accesses the same live project through the connected WebMCP session. The agent does not need direct MIDI hardware access: it reads the resulting notes and project state, then proposes or applies visible, undoable musical edits.

This separation keeps the human performance layer local and hardware-friendly while preserving the agent’s ability to collaborate through the browser.

Challenges I ran into

The core challenge was not simply generating musical material—it was making agent actions feel safe and legible.

I designed the tools around a few principles:

1. The agent receives compact state summaries before acting, rather than a flood of raw notes.
2. Music theory remains in pure deterministic functions, so harmony does not depend on model guesswork.
3. Write operations validate their inputs and preserve an inverse patch for Undo.
4. The person can see what changed, why it changed, and which bars were affected.
5. Drafts let the person hear alternatives before committing them.

I also had to balance expressive composition tools with small, precise schemas that are useful to WebMCP clients.

Accomplishments that I'm proud of

• A live browser session where a person and AI can compose on the same page.
• WebMCP tools that support both reading context and making safe musical edits.
• Agent-authored notes are visually distinct from human-authored notes.
• Every AI mutation has a human-readable explanation and one-click Undo.
• Deterministic harmony, voicing, bass generation, and mode transformations.
• No backend, authentication, analytics, API keys, or remote project storage.
• An end-to-end test: read a person’s notes, add accompaniment, undo it, then reapply a different arrangement through WebMCP.

What I learned

I learned that agentic interfaces become much more trustworthy when they expose small, contextual tools instead of treating the app as a black box.

In creative software, reversibility is not a safety feature added at the end—it is part of the creative experience. Clear diffs, explicit intent, deterministic musical rules, and visible state make it easier for a person to stay in control while still benefiting from an AI collaborator.

What's next for Duet

Next, I would like to add richer MIDI export/import workflows, more arrangement patterns, stronger visual auditioning for draft variations, and broader browser support as WebMCP availability grows.

Acknowledgements

Special thanks to my wife, a pianist and piano teacher, who performs in the demo video. Her musician’s perspective, feedback, and willingness to test the human side of the workflow were essential to shaping Duet. This application would not have been possible without her.

Built With

• document.modelcontext
• react
• tonal
• tone.js
• tonejs/midi
• typescript
• vercel
• vite
• webmcp
• webmidiapi
• zustand