SightSinger Copilot

Say the edit, hear the change. WebMCP fuses SightSinger and ChatGPT into one AI co-composer, precise score edits in plain language, and sing it back instantly. No DAW required.

Comment

Inspiration

SightSinger is a production score-to-singing tool with initial traction. Since launching in May 2026, it has reached 600 sign-ups, 200 active users, and MRR enough to cover my AI coding subscription. Our most requested features are a one-click "render all parts" button and cleanup of common score artifacts like redundant notes and misaligned lyrics.

Both asks are harder than they sound. "Render all" means reliably orchestrating many long-running synthesis jobs. Score cleanup means running arbitrary code against user data. It requires a real sandbox to lock down which libraries and tools that code can touch. That's a security and infrastructure investment in its own, unrelated to the singing engine SightSinger was built to be good at.

WebMCP solved the problem. Rather than building all by myself, I could expose only what SightSinger uniquely does well, like precise per-part synthesis and expressive singing, as narrow, callable tools, and let the requesting agent, already running under its own host's sandboxing and orchestration, handles the rest. The dangerous, generic work moves to systems already built to handle it.

This isn't "let's add WebMCP because it's trending." The result is SightSinger Copilot: an edit, see, sing loop, where the agent handles precise editing and sequencing, and SightSinger handles the voice.

What it does

SightSinger Copilot turns a written MusicXML score into an agent-ready rehearsal studio. Instead of prompting for a generic song, musicians work from the notes, rhythm, lyrics, parts, verses, and instrumental arrangement already written in their score.

A user can upload a score, or start with the built-in Happy Birthday demo, then ask an AI agent to inspect the score, make precise MusicXML edits, generate solfège lyrics, direct a vocal performance, synthesize a selected vocal part, and export the current vocal mix. The resulting vocal tracks sit alongside score-derived instrumental MIDI playback, so users can rehearse and iterate without DAW editing.

WebMCP makes this workflow agentic and beyond a commodity score-to-singing tool. A musicXML score is structured text, any edit precise enough to describe, like "at bar 15, change the chord from D minor to F major 7th," becomes an exact change, not a lucky regeneration.

How we built it

We built an open-source React/TypeScript frontend hosted on Firebase Hosting. The frontend registers score-aware WebMCP tools at startup, letting an agent retrieve and edit the active MusicXML, add solfège, request synthesis, check credits, and export the current mix. The browser UI remains the visible workspace for score updates, rendering progress, multitrack playback, and downloads.

A separate private Cloud Run backend handles authorization, score/session storage, and GPU-backed DiffSinger inference. It runs on an NVIDIA L4 GPU and turns the selected score part, lyrics or solfège, and performance settings into vocal tracks that return to the studio for playback and mixing.

Challenges we ran into

WebMCP tools do not currently provide a native way for an agent to attach a local file to the site or directly receive a downloaded audio file. That matters for a score-first music workflow.

We designed around this with URL- and session-based handoffs. The musician uploads the initial score in the studio, then the agent retrieves and edits the active MusicXML through the current session.

Synthesized tracks and exported mixes are returned as playback/download URLs, while the studio updates the visible score and multitrack player. This keeps every agent action grounded in the same active session without requiring unsupported file-transfer capabilities.

Accomplishments that we're proud of

I discovered this challenge only about two days ago, and the public repository was created just for it. In that short time, we learned WebMCP, exposed meaningful SightSinger capabilities as browser tools, and delivered a working, deployable demo.

I'm proud that this is more than a tool-call showcase. It demonstrates a practical score-first workflow: an LLM can understand the active MusicXML score, make natural-language edits, prepare solfège or lyric renders, direct an AI singer, and help assemble a rehearsal-ready demo. All while keeping the musician in the visible studio loop.

What we learned

Before writing code, I asked which agent-integration approach actually fits: vision/screenshot control, DOM scraping, A2A, a hosted MCP server, a direct API, or WebMCP.

WebMCP stood out for two reasons. It calls structured tools rather than clicking rendered pixels, so it's way more token efficient and accurate. And it runs the agent directly on our page, inheriting the user's existing logged-in session, no separate server, auth, or API keys to stand up. That also meant the riskiest part of this feature, an LLM freely editing a user's score, runs inside the agent's own sandbox instead of one we'd have to build ourselves.

The lesson: there is no universally right integration approach. Only the approach that best fits the use case, constraints, and time.

What's next for SightSinger Copilot

Next, we want SightSinger Copilot to become a true co-director for score-based vocal production. It should understand a song’s musical context, including its form, phrasing, lyrics, harmony, vocal range, and emotional arc. It then translates that understanding into precise performance direction for the AI singer.

For example, Copilot could direct: “bars 1–4: soft, peaceful, and clean,” “bars 5–8: gradually build intensity,” and “chorus: high-energy, emotionally open.” SightSinger would then translate the director’s or composer’s instruction into its voicing-curve parameter controls, shaping the singer’s delivery across the score. Rather than treating a whole part as one uniform render, SightSinger could create vocals that remain faithful to the written notation while carrying intention, contrast, and emotion.

Built With

Share this project:

Updates

Submission history