HARMONIUM ( AI PLAYS MUSICAL INTRUMENTAL)

poster
GIF
agent play
GIF
human play
banner

Inspiration

codex plays HARMONIUM ( traditional indian musiscal isntrument - similar to a piano )

I was listening to piano music while my team and I were working on Draw Generator, another project for the same hackathon. The music gave me a glimpse of my childhood memories with the harmonium, and that moment gave me this idea.

I started wondering: CAN it actually PLAY AN MUSICAL INSTRUMENT with us? and How does AI understand music, and can We tested this idea with Codex using GPT-5.6, and the result was wonderful. That experiment became HARMONIUM.

What it does

HARMONIUM is a playable 39-key Indian harmonium covering C3 to D6. People can play it using a mouse, touchscreen, or computer keyboard.

It supports Western and Sargam notation, lets the user move the Sa reference, loads structured JSON scores, and displays falling notes that remain synchronized with the music.

The main idea is simple: people and AI agents can play the same visible instrument together. When an agent plays a note, the human can hear it, see the matching key light up, and continue playing from the same instrument.

How we built it

We built HARMONIUM with React, TypeScript, Vite, Tone.js, the WebAudio, and WebMCP.

The page exposes five WebMCP tools:

play_note plays one note.
play_score performs a timed melody or chord sequence.
set_sa changes the Sargam reference.
stop ends the current performance.
get_state lets the agent inspect the instrument.

Agents and people use the same visible instrument. The tools are registered directly on the page using document.modelContext.registerTool().

A single audio engine owns all note playback. Mouse input, computer-keyboard input, JSON scores, and WebMCP tools all call the same playNote and stopNote logic. This keeps the sound and visible keyboard synchronized instead of maintaining separate implementations for humans and agents.

Tone.js schedules the notes using the browser's audio clock. The instrument uses real harmonium samples, while a synthesized reed-like sound acts as a fallback if the samples cannot load.

Challenges we ran into

The biggest challenge was understanding how sound works on this instrument. We had never worked with music on the web before. As we started, we got help from a teacher from my childhood, and this gave us a real boost.

ChatGPT also did a great job helping us research how the harmonium works, understand browser audio, find useful resources, and build the first working version.

Keeping sound and animation synchronized was another challenge. JavaScript timers can drift, especially during longer scores, so we used Tone.js Transport and the Web Audio clock for playback and visual timing.

Browsers also block audio until the human interacts with the page. An AI agent cannot unlock the audio by itself, so the user must first press Tap to start. After that, the agent can play notes and complete scores through WebMCP.

Accomplishments that we're proud of

We are proud that HARMONIUM looks and feels like a complete instrument rather than a WebMCP demonstration attached to a webpage.

The project includes real harmonium samples, synchronized visuals, structured tool results, and WebMCP evaluation fixtures for tool selection and multi-step journeys.

A person can play manually, hand control to an agent, change the note or Sa reference, and then continue playing. Agent actions remain visible to the human.

This synchronizes human mode and agent mode through one shared implementation. That made WebMCP a genuine extension of the product instead of a separate feature added only for the hackathon.

We are also proud that every interaction uses the same audio engine. A note sounds and lights up in the same way whether it comes from a mouse, computer keyboard, JSON score, or AI agent.

What we learned

We learned how music works inside a web browser. Before this project, we had never built a musical instrument for the web. We learned about notes, timing, overlapping sounds, audio samples, the Web Audio clock, Western notation, and Indian Sargam notation.

We learned that synchronizing music is different from updating a normal interface. A small delay between the sound and the highlighted key is immediately noticeable. Using Tone.js Transport and the Web Audio clock helped us keep the notes, falling visuals, and keyboard state together.

We also learned why one shared implementation matters. Because people and agents use the same audio engine, we do not need separate logic for human mode and agent mode. This made the behavior easier to understand and prevented the two modes from becoming different instruments.

WebMCP taught us that an agent should not have to guess where to click on a webpage. Instead, the website can expose clear actions such as play_note, play_score, and set_sa. The agent gets a reliable way to act, while the human can still see and control everything through the normal interface.

We learned that tool descriptions, input validation, error messages, and state inspection are important parts of the agent experience. For example, if audio is still locked, the tool must clearly explain that the human needs to press Tap to start.

Most importantly, we learned that AI does not have to replace the person using an application. In HARMONIUM, the agent and the human share the same instrument. One can begin a musical idea, and the other can continue it.