Agent Riff

Agent Riff — Play a Musical Idea Together
Live Musical Conversation in the webpage
Human Plays the First Phrase
Agent Responds with Its Own Phrase

Life in Tech is already complicated, so many tools and frameworks to learn each day and tons of stress to deal with. So why not add more stress to your life when you find out your agent is better than you at playing music!!

Inspiration for this Project

Most AI music tools work in roughly the same way: you describe what you want, wait for the model, and receive a finished piece of music.

We wanted to try somethingball that felt more collaborative. Instead of asking an AI to make music for us, we asked a different question: could a browser agent actually play music with us?

That idea became Agent Riff, a musical call-and-response experience between a person and their browser agent. The person plays a short phrase, the agent receives the notes and timing through WebMCP, and then responds with a phrase of its own.

The goal was not to create another song generator. We wanted the agent to feel like a second musician who listens before deciding what to play.

What it does

Agent Riff is a playable browser instrument built for live human and AI collaboration.

A person can play notes using a mouse, touchscreen, or the keys on their computer keyboard. The application captures the performance as structured musical events, including the notes, timing, tempo, key, scale, instrument, register, and density.

The browser agent can then:

Join the current session
Analyze the latest human phrase
Choose a complementary instrument and register
Perform a precisely timed response
Wait for the person to play another phrase
Change the shared tempo, key, or scale when requested
Perform longer musical sets efficiently

Agent Riff does not use the microphone or record the user's audio. The agent receives the musical information already created inside the application. This gives it accurate notes and timing without asking it to interpret sound or guess what happened from a screenshot.

Human and agent performances appear in a shared live score. Each completed phrase is also saved as a clickable card. A person can replay their own phrase, replay the agent's response, or play both simultaneously to hear how well they work together.

The agent can remain available for up to ten minutes between turns. This means the person can continue playing without repeatedly returning to the chat and typing "listen again." If they want to stop sooner, they can release the agent directly from the interface.

How we built it

Agent Riff is built with vanilla JavaScript, HTML, and CSS. All instrument sounds are synthesized directly in the browser using the Web Audio API.

There are no recorded samples, user accounts, external music services, or runtime dependencies. A person can open the page and begin playing immediately.

The application exposes seven WebMCP tools through document.modelContext.registerTool():

riff_join_session
riff_get_session_state
riff_listen
riff_wait_for_human_phrase
riff_perform_phrase
riff_perform_set
riff_set_compass

These tools give the agent musical abilities rather than generic browser controls.

For example, the agent does not need to find individual piano keys on the screen and click them at the correct moment. It can call riff_listen to receive an analysis of the recent performance, then call riff_perform_phrase with the notes, durations, timing offsets, instrument, and velocity of its response.

The performance tool supports sequential notes, rests, chords, overlapping notes, and exact beat-relative timing. For speed, the agent can use a compact score format instead of sending a large object for every note.

The waiting tool keeps the call-and-response session active. When the person finishes another phrase, the tool returns both the new notes and the analysis needed for the next response. This lets the agent perform directly without making an additional listening call.

We also created riff_perform_set for requests involving complete songs or multiple sections. It can schedule several public-domain or custom pieces in one call, reducing repeated model and browser round trips.

Every tool uses a bounded JSON Schema, and all inputs are validated again during execution. The project also includes automated tests for musical analysis, input constraints, WebMCP registration, tool behavior, cleanup, and the production deployment artifact.

Challenges

The first version technically worked, but it did not feel much like a live musical interaction.

The agent would inspect the session, analyze the notes, think about its response, call another tool, play a very short phrase, and then return to the chat. The delay between turns was long enough that it was easy to forget what had just been played.

We reduced that delay by combining related information. The waiting tool now returns the completed phrase together with its musical analysis and a suggested response shape. We also added compact score notation so a complete response can be sent in one efficient tool call.

Response quality created another tradeoff. Faster models replied more quickly, but their musical answers were often only a few notes long or sounded like random keyboard input. More capable models produced better phrases, but took longer to respond.

We improved this by giving the agent clearer musical guidance. Normal responses are encouraged to develop across roughly 12 to 16 notes, with an echo of the human idea, a variation, and a resolution. Dense human phrases receive shorter or more spacious answers so the agent does not overcrowd the music.

We also had to account for ChatGPT's side-by-side browser layout. A design that looked good in a full browser window became extremely tall and difficult to use when the browser occupied only half the screen. We redesigned the responsive layout so the instrument, score, activity, agent status, and copied prompt remain accessible in that narrower configuration.

Long waits created a separate usability problem. People needed a clear way to tell whether the agent was still present and how to stop it. We added visible agent status, a bounded waiting period, and a release button that ends the current wait and cancels queued agent playback.

Finally, browser audio has its own restrictions. Audio must begin from a user interaction, and keyboard input should not interfere with focused form controls. We had to balance reliable instrument input with normal keyboard and accessibility behavior.

What we learned

The biggest lesson was that a good agent experience depends just as much on tool design as it does on the model.

When an agent receives vague page content and generic browser controls, it spends much of its effort figuring out how to operate the interface. When it receives focused tools with compact, structured context, it can spend that effort making a better creative decision.

We also learned that fewer tool calls do not necessarily mean less capable tools. Combining the right information at the right moment made the experience faster while giving the agent more useful musical context.

WebMCP was a natural fit for Agent Riff because the important information is difficult to communicate through pixels alone. A screenshot can show a piano, but it cannot reliably describe the exact notes, timing, density, and musical direction of the phrase that was just played.

With WebMCP, the website becomes more than an interface the agent operates. It becomes a shared creative space where a person and an agent can listen, respond, and build something together.