Euterpe

Inspiration

I wanted a middle ground between learning a full music-production application and asking a generator to make a finished song. A beginner may already have the important part — a line they can hum or sing — but not yet know how to turn it into chords, a bass part, drums and an arrangement. The useful collaborator is not one that replaces that idea. It is one that can work on the same song, explain a choice, let the person hear it, and accept “undo” as a normal part of making music.

Music also exposes why WebMCP is useful. The page owns the microphone, song document, audio clock and speakers. The person is the only one who can hear whether a change works, while the browser’s agent needs exact notes, chords, bar ranges and revision state rather than a screenshot. WebMCP gives the site a structured way to expose those actions to the agent, as described in OpenAI’s Site tools guide and Chrome’s WebMCP guide.

What it does

Euterpe opens as an empty eight-bar song with one melody track. A person can record a hum or sung line, import a browser-decodable voice memo, play the musical-typing keyboard, or use a MIDI input where the browser supports it. Every path produces a take with rough symbolic notes; microphone and imported takes also retain the original audio.

The browser’s agent has 29 registered tools. It can read the song, notes, chords and takes; suggest or set chords; offer two or three alternatives that the person can audition before choosing; ask the person to perform a named part over particular bars; set the key and tempo; add tracks; generate deterministic chord, bass and drum parts; arrange sections; mix, play, stop, undo and redo; and start cancellable WAV, MP3 or MIDI export jobs. The five track kinds are melody, chords, bass, drums and vocal.

The vocal is part of the song rather than disposable input to pitch detection. A dedicated vocal track can hold the retained recording. tune_vocal applies reversible, key-aware pitch correction, set_quantize aligns its timing with adjustable strength and optional swing, and set_mix controls level, pan, mute and solo. WAV and MP3 renders include retained voice clips. MIDI is notes-only, so Euterpe refuses MIDI export when the song contains retained audio instead of silently dropping the singer.

The agent does not hear the audio and the tool boundary does not send it audio. It receives bounded JSON descriptions of musical state; the person remains the ears and makes the final choice. Every newly authored musical change carries a plain-language why, appears in the activity strip and producer notes, points back to the bars it changed, and can be undone. The agent also cannot grant microphone permission or bypass the browser’s audio rules for the person.

How we built it

The application is a TypeScript and React client built with Vite and hosted as a static site. There is no model or music-generation service inside the page. The browser’s agent chooses the tools; deterministic local code handles state, music theory, pitch analysis and audio.

All human and agent edits go through one serialised command bus and one song reducer. A successful edit increments a monotonic revision and returns the changed fields, summary and target bars. Tools can include an expected revision, so a stale write is refused with a short account of what changed instead of overwriting newer work. Human drags on the piano roll briefly hold agent writes, and a track being recorded is locked while other tracks remain editable.

Each tool has a Zod input schema exported as strict JSON Schema, a short description, annotations and an execute function. The registry installs the same 29 definitions through document.modelContext.registerTool and the compatibility navigator.modelContext path, deduplicating them when both names refer to the same context. Six tools are marked read-only. Results use one compact success-or-error envelope and stay within the application’s 1,500-character budget.

The audio path uses Tone.js and smplr for scheduled instruments, with an AudioWorklet and Pitchy for live and offline pitch tracking. Tonal supplies note and chord primitives. Recorded mono PCM is stored with the take, referenced by clips on the timeline, and scheduled through the same vocal path for live playback and offline rendering. WAV is encoded as PCM; MP3 work runs in a separate worker through Mediabunny; MIDI is written from the symbolic tracks.

Recording over an arrangement uses a shared worklet clock and the scheduled count-in boundary, then combines those with the active input device’s reported latency and the audio context’s base and output latency. These are browser and device estimates under the Media Capture and Streams and Web Audio APIs, not a claim of acoustic sample-accurate alignment. If the input device does not report latency, Euterpe refuses microphone capture rather than inventing an offset.

At the submission commit, the unit and integration suite passes 817 tests across 110 files. The browser harness contains 126 asserted steps across seven scenarios, covering the 29 tool behaviours, revisions, error envelopes, recording locks, retained-vocal alignment, real audio export and sample fallback.

Challenges we ran into

The largest mistake in the first version was architectural: a hummed take was analysed into notes and then the recording itself was thrown away. That was acceptable for a transcription demo but wrong for a music maker. Keeping the singer meant adding serialisable PCM to the take model, clip references to tracks, persistence and history support, clip copying during arrangement, a separate live-audio path for vocal tracks, and the same path again in offline WAV and MP3 rendering.

Recording over music was another real boundary. The capture worklet, count-in and output device expose related but different timing information. A fixed “latency compensation” number would have looked convincing on one machine and been wrong on another. The implementation now records the worklet boundary, scheduled arrangement boundary and the browser-reported input, processing and output estimates with each microphone take. Missing or non-monotonic evidence causes a refusal, not a guessed placement.

Making vocal changes reversible also took more than moving note rectangles. The original note timing has to survive quantisation, while the retained PCM needs a corresponding time map. Pitch correction has to use voiced pitch frames and the current key without changing the stored recording. Both operations therefore live as adjustable clip settings in the song history rather than destructive rewrites of the source audio.

Export forced an explicit product decision. A Standard MIDI File can represent the note tracks but not the retained PCM recording. Silently exporting the instruments would produce a valid file that was not the song the person heard. The encoder and WebMCP tool both reject that case and direct the person to WAV or MP3.

Finally, WebMCP’s structured boundary cannot listen for the agent. Read tools therefore return small musical facts — notes, chord symbols, bar ranges, take quality and revision — while the page makes every action visible and leaves listening to the person. That constraint shaped the product more than adding a chat panel would have.

Accomplishments that we're proud of

The singer survives the whole journey. A hummed or sung line is recorded, retained as audio, tuned to the key, aligned to the grid, mixed, and exported inside the finished WAV or MP3 — while the same take also drives the symbolic notes the agent reasons over. Both representations stay in one song document with one history.

Euterpe refuses in three places where it would have been easier to guess. It refuses microphone capture when the input device will not report its latency, rather than applying an offset that looks right on one machine and is wrong on another. It refuses MIDI export when the song contains retained audio, rather than producing a valid file that is not the song the person heard. And it refuses a stale write when the revision has moved, rather than overwriting newer work. Each refusal says what happened and what to do instead.

The agent is genuinely useful without ever hearing the music. Twenty-nine tools give it notes, chords, bar ranges, take quality and revision state as bounded JSON; it proposes, explains and edits; the person listens and decides. Every authored change carries a plain-language reason, points at the bars it touched, and can be undone.

The evidence is real rather than asserted: 817 tests across 110 files, and a browser harness of 126 asserted steps in seven scenarios covering every registered tool, the revision and locking rules, retained-vocal alignment and actual audio export.

What we learned

WebMCP is strongest when a site has live state or local capabilities that should not be moved into a remote service. Here it lets the agent work inside the instrument that is already recording and playing, while the person can take over at any point.

I also learned that a useful tool surface needs concurrency rules, bounded reads, explicit failures and visible consequences, not just a list of functions. The revision check, serial queue, recording lock, producer notes and per-change undo are as important as chord generation.

Most importantly, pitch detection is evidence rather than musical truth. A rough hum can contain breath, slides and repeated detections that should not automatically become the arrangement. Offering interpretations, preserving an untouched “keep what I sang” choice, and retaining the actual voice produced a more honest collaboration. The project does not claim a trained local model or a model-quality benchmark: the page uses deterministic analysis and generation, the browser’s agent reasons over the results, and the person judges the sound.

What's next for Euterpe

Replace the device-reported latency figures with an optional acoustic loopback calibration, so alignment can be measured on the person's own hardware instead of trusted from the browser. The current estimates are honest but coarse, and the refusal path exists precisely because they can be absent.

Support more than one vocal take per part, with comping across takes, so a singer can keep the best phrase from each attempt in the way a real session works.

Generate harmony and doubling parts from the retained vocal rather than only from the symbolic notes, using the pitch frames already captured.

Widen what the agent can reason about structurally — sections, repeats and song form — so the conversation can move from "what chords" to "what should happen next".

Two limits stay deliberate. The agent will not receive the audio itself; the person remains the ears. And Euterpe transcribes pitch and timing, not lyrics, so the words are heard rather than read.

Built With

• audioworklet
• chrome-devtools-mcp
• cloudflare-pages
• cloudflare-r2
• css
• html
• javascript
• media-capture-and-streams-api
• mediabunny
• music
• node.js
• pitchy
• react
• smplr
• tonal
• tone.js
• typescript
• vite
• vitest
• web-audio-api
• web-midi-api
• webmcp
• zod