MuseFlow

Inspiration

ChatGPT can suggest a chord progression in text, but it cannot hear the one you just played. Existing AI music tools generate music from scratch but don't collaborate on music you're already making. They don't know your key, your tempo, or the fact that you just stopped playing mid-phrase and are waiting for an idea. We wanted to close that loop — to create an AI that reads your live audio and answers with harmony you can hear.

What it does

• Real-time chord detection: Analyzes your live audio to identify chords, inversions, and key context
• AI-powered harmony suggestions: Suggests functional chord progressions and melodic continuations
• Dynamic tool surface: Tools appear and disappear based on your playing (e.g., "analyze-phrase" appears when you finish a phrase)
• Multiple input methods: Play via mouse, touch, typing keyboard, or MIDI controller
• No microphone required: Reads the audio graph directly from your browser
• Instant feedback: The AI can play back suggestions through the same audio engine

How we built it

• Framework: Next.js 14 with TypeScript (App Router)
• Audio: Web Audio API — native oscillator bank with ADSR envelopes
• DSP: Hand-rolled algorithms — YIN for pitch detection, chromagram + template matching for chords, spectral flux for tempo
• Music theory: Custom implementation of functional harmony, Roman numeral analysis, and key detection (Krumhansl–Schmuckler)
• AI integration: Anthropic Claude API via WebMCP document.modelContext for tool registration and execution
• Tool surface: 12 WebMCP tools (10 static, 2 dynamic) that expose audio capabilities to the agent
• Polyfill: Spec-shaped WebMCP polyfill for standalone operation when external agent not present

Challenges we ran into

• Web Audio API limitations: The audio graph is browser-local — no server can reach it, requiring WebMCP to bridge the gap
• Real-time DSP performance: Implementing YIN pitch detection and chromagram analysis in JavaScript without external libraries
• Dynamic tool registration: Managing tools that appear/disappear based on audio events (e.g., phrase completion)
• Cross-network agent loop: Splitting the agent execution between browser (tools) and server (API) while maintaining statelessness
• Polyphonic pitch detection: YIN is monophonic by design — we had to handle polyphonic input gracefully by redirecting to chord detection

Accomplishments that we're proud of

• First-of-its-kind integration: Successfully bridged browser-local audio with AI agents using WebMCP
• Hand-rolled DSP accuracy: Achieved 440.02 Hz, 0 cents off, confidence 1.0 on sustained A4 through our custom YIN implementation
• Dynamic tool surface: Implemented tools that rewrite their descriptions based on what you actually played (e.g., "Respond to F#3")
• No external audio dependencies: Built everything from scratch — no Tone.js, no teoria, no pitchfinder
• Privacy-first design: No microphone, no recording, no upload — everything stays in your browser
• Working demo: Created a hands-free /demo walkthrough that showcases real tool calls and audio analysis

What we learned

• WebMCP's unique value: It enables capabilities that are literally impossible otherwise — browser-local state cannot be reached by any server-side API
• DSP in JavaScript: Real-time audio analysis is feasible in the browser with careful implementation
• Agent-tool interaction patterns: Dynamic tools that respond to user behavior create more natural collaboration
• Split architecture: Separating tool execution (browser) from model inference (server) is the right pattern for browser-local capabilities
• Music theory implementation: Building functional harmony from scratch requires deep understanding of chord progressions and voice leading

What's next for MuseFlow

• More instruments: Add guitar, bass, and drum kit support with appropriate DSP
• Advanced harmony: Implement jazz extensions, modal interchange, and voice-leading suggestions
• Recording & export: Allow users to record sessions and export MIDI/audio
• Collaborative sessions: Multi-user jam sessions where multiple agents can participate
• Mobile optimization: Better touch support and mobile-specific UI
• Additional models: Support for other AI providers beyond Anthropic

Built With

• webmcp