Default startup interface
Click on play and see the synth in action.
Message sent to live AI agent.
WebMCP tools verified.
Confirmation of agent not changing things manually locked by hand.
One single source of truth.
Self-testing.

Inspiration

Every AI-powered creative tool right now works the same way: you describe what you want in a chat box, and the model either generates a static result or silently mutates state you can't see happening. Sound design tools especially suffer from this — either the agent is a chatbot bolted onto the side of a DAW, guessing its way through a UI it wasn't built to understand, or it just spits out a finished audio file with no room for you to steer it mid-process.

WebMCP flips that. Instead of an agent scraping a UI, the app itself declares exactly what it can do — real, typed, callable tools. That felt like the right foundation for something more interesting than "agent generates a sound": a studio where a human and an agent operate the same live instrument at the same time, each able to see and change what the other just did.

What it does

Mysynote is a browser-based modular synthesizer built on the Web Audio API. You patch together oscillators, filters, distortion, delay, reverb, and a sequencer on a live canvas, exactly like a hardware modular rig. An AI agent (Gemini or Claude, bring your own API key) sits alongside you with access to the same graph through WebMCP tools — it can inspect the current state, spawn and remove modules, patch and unpatch cables, tweak parameters, and read the live spectrum, all through structured tool calls rather than by describing changes in text.

The part I care about most: you can lock any parameter by hand. Once locked, that boundary isn't a suggestion sitting in a system prompt the model could talk itself around — it's enforced in the tool implementation itself. If the agent (or a WebMCP-enabled browser calling the tools directly) tries to write to a locked parameter, or even tries to call unlock_parameter on a lock you set, the call is rejected with a structured error before it ever reaches the audio graph. You define the boundaries; the agent works freely inside them.

How I built it

The frontend is Next.js/React/TypeScript, styled with Tailwind. The audio engine is hand-written against the raw Web Audio API — no framework wrapping it — with a reconciler that diffs a plain-object graph state against the live AudioNode graph and patches only what changed, so the agent's tool calls and your own dragging/clicking stay in sync against one authoritative source of truth instead of two.

WebMCP tools are registered via document.modelContext.registerTool: get_audio_graph_state, spawn_module, remove_module, patch_cable, remove_cable, set_module_param, lock_parameter, unlock_parameter, get_spectrum_analysis, get_sequencer_state, set_sequencer_step, and clear_graph. The in-app agent console (Gemini/Claude, BYOK) drives these exact same tools, so what you see the agent do in-app is identical to what any WebMCP-aware browser or agent could do against the live site.

Challenges I ran into

The hardest bug wasn't a crash — it was a bug that looked like everything worked. Early on, a locked parameter really did block direct writes, but the agent could still call unlock_parameter on itself with no restriction, defeat the lock, and then set the value anyway. Nothing crashed; it just quietly broke the entire premise of the project. Fixing it meant tracking who set a lock (human vs. tool) and rejecting a tool-sourced unlock of a human-set lock at the enforcement layer — plus adding it as a permanent regression check in the app's own self-test, so it can't silently reappear.

I also hit a nasty reconciler bug where removing a module threw an uncaught exception mid-cleanup (a plain string field was being treated like an AudioNode), which left orphaned nodes that kept crashing every subsequent state change. Chasing a bug that hides behind unrelated symptoms — a stuck test button, a sequencer step failing — down to one bad property check was the toughest debugging of the project.

What I learned

Building the "hard boundary" wasn't a UI problem, it was an authorization problem — the interesting design work in an agent-native app isn't the tools you expose, it's deciding which of your own tools the agent is allowed to call on itself. And that a feature can look completely correct in every manual test and still have a real security-shaped hole in it until something actually tries to defeat it on purpose.

What's next

Multiplayer patching (multiple humans + one agent on the same graph), saved/shareable patches, and exposing the sequencer's per-step modulation as its own lockable, agent-modifiable surface.