Jam | Devpost

You know the skit — someone makes a dumb sound with their mouth, and it turns into an actual song. We've all watched it and thought "I bet you could actually build that." When OpenAI dropped the WebMCP hackathon, that was the excuse to stop thinking about it and just build it.

What it does

Jam is a browser-based sampler/mini-DAW where you record a sound — your voice, a beatbox, literally anything into the mic — and it becomes a real, playable instrument. From there, an AI agent can take over through WebMCP tools registered directly on the page:

capture_sound — records a short clip from the mic create_sampler_node — turns that clip into a track map_sample_to_keys — pitch-maps it so it plays melodically across a keyboard play_note — plays specific notes on any track generate_melody — writes a short call-and-response pattern onto a track instead of a flat repeating loop suggest_next_layer / add_complementary_node — the agent looks at what's missing (no bass? no drums?) and proposes and builds a complementary layer transform_instrument, mute_node, set_tempo, move_node — the rest of the mixing/arranging toolkit get_studio_state — lets the agent see what nodes already exist before acting on them

So the flow is: say something goofy into the mic, and watch an agent turn it into a full arranged track, live, calling real tools in the browser the whole time.

How we built it

React for the UI, the Web Audio API for everything sound-related (synth engines, sample playback, pitch-shifting via playback rate, a live transport/timeline), and navigator.modelContext.registerTool (WebMCP) to expose the tools an agent can call.

We used ChatGPT and Codex as part of the actual build process — wrote a PRD first to nail down scope before any code, then let Codex build against it, then iterated by hand on top of that (the WebMCP wiring, the melody generator, the touch-friendly piano, the visual polish).

Melody generation went through a real revision: the first version was a random walk that sounded mechanical and repetitive. It's now motif-based — a rhythm template split into a "call" phrase and a "response" phrase, where the response either echoes the call transposed or, when answering another track, deliberately avoids that track's onsets so two layers interlock instead of colliding.

Challenges we ran into Stale closures. WebMCP tools get registered once on mount, but the tool implementations need the current app state every time they're called — not whatever state existed at registration time. Fixed by routing tool execution through refs that stay in sync with state, instead of closing over state directly. Making generated music not sound like a robot. Uniform random-walk melodies technically worked but sounded lifeless. Moving to rhythm templates + call-and-response + per-note velocity humanization + phrase-ending sustain notes made a real audible difference. Recording UX. Press-and-hold recording is fragile — it breaks on touch devices and on any stray pointer movement off the button. Switched to a simple tap-to-start/tap-to-stop toggle. The on-screen piano. Mouse-only event handling meant it didn't work properly on mobile at all. Rebuilt on Pointer Events (unifies mouse/touch/pen) with real glissando support — press and slide across keys like an actual mobile piano app — plus a global pointer-up listener so a dragged-off touch can never leave a key "stuck." Punchy samples from a raw mic recording. Voice memos recorded straight from getUserMedia tend to have dead air and inconsistent loudness, which sounds especially bad once you're pitch-shifting them across a keyboard. Added a trim/normalize/fade pass on every captured clip before it becomes a track. Accomplishments we're proud of

Every tool an agent can call is a real, functional action against the live app state — not a stub. You can watch an agent build a track from nothing but a beatboxed sound in under a minute, live, with the tool calls visibly firing in an on-screen activity feed.

What we learned

That the hardest part of "AI makes music" isn't wiring up the tool calls — it's making the generated output not sound like a demo. Musical structure (rhythm templates, phrase shape, call-and-response) mattered far more than raw randomness ever could.

What's next Onset-detection on recorded samples so generated melodies can actually match the rhythmic character of what you recorded, not just a generic scale More instrument variety once the current set is dialed all the way in Exporting the finished loop as an actual audio file Content jam-ui (1).jsx

JSX

jam-ui (1).jsx

JSX