Air Instruments

• GIF

  air-instruments

Inspiration

Built in one day by an ex-StreamYard engineer and guitar player whose mouse broke this week — so fingers became the instrument. The sticks are virtual because "real sticks are my index fingers."

The deeper thesis: an agent that teaches an instrument should read the state that actually matters — the notes you played, their timing, your score — never pixels, and it should never apply its own proposal without you.

What it does

Drums, a 25-key keyboard, and a six-string guitar (electric or acoustic) live in a single web page. A webcam tracks your hands; pinch and strum, tap and pluck, or just use a mouse, touch or keyboard — this isn't camera-only.

Learn music: 27 short lessons across all three instruments, with a local, rule-based teacher that scores your timing note-by-note.

Learn with an agent: the page registers seven tools on document.modelContext (WebMCP). An agent in the browser can read your completed take, open a lesson, demonstrate it on the real instrument, point at the exact step you fumbled, explain why in its own attributed panel, and propose an easier exercise. A human presses Apply exercise or Keep mine — the agent can never apply its own proposal — and Undo agent exercise restores your prior state. The panel shows recent tool calls so you can watch exactly what the agent touched.

This whole loop is self-contained in the page — no backend, no microphone, works for anyone who opens the live URL.

Why WebMCP is a good fit

An agent can already talk about music. It cannot hear you play, and it cannot reach into a page to open the right lesson. Everything that matters — the note the audio engine actually produced, when it landed, whether it was on the beat, which lesson is open, what your last take scored — lives in the page. That state is worthless in a chat box and priceless to a teacher.

There's no server to run, no API key, no MCP endpoint to host. The page registers seven tools with document.modelContext and any WebMCP-aware agent in that browser can use them. Camera frames and microphone audio are never in a tool result — there's no microphone in this build at all.

How we built it

Plain ES modules, no build step, no framework. Seven tools via document.modelContext.registerTool (with a navigator.modelContext fallback): reading state, listing and opening lessons, playing a demo or a take, proposing and explaining exercises, and focusing attention on one step. Local hand tracking runs in a worker; Web Audio does the drum, key and string synthesis, including a real plucked-string model for guitar. Every agent proposal carries a revision check — change the exercise yourself and a stale proposal is rejected rather than silently applied.

Challenges we ran into

• The guitar's pinch-and-strum input had a real coordinate-mapping bug: the code only registered a strum inside a small active zone of the stage, so a hand could be visibly crossing the strings and still trigger nothing. Traced and fixed.
• Getting the same teach → try → feedback → easier-exercise loop to work identically across three very different instruments (drums, keys, strings) took more care than expected — each has its own timing and input model underneath one shared agent contract.

What we learned

Building a teaching loop where the agent can suggest but never act is a different design problem than a normal tool-calling agent — the human has to stay the one holding the instrument, in the interaction design itself, not just in a permissions check.

What's next

• More instruments on the same agent contract
• Multiplayer lessons, so two people (or a person and an agent) can play the same piece together

Tool-calling verified in Chrome with WebMCP enabled. The in-page agent flow inside ChatGPT's own in-app browser has been used live to test and debug the guitar input, as shown in the submitted image gallery.

Built With

• javascript
• mediapipe
• webaudioapi
• webmcp