Overture

Inspiration

Learning an instrument alone is lonely and slow. You squint at sheet music, fight a metronome, and never really know whether you rushed the hard bar or just think you did. Most practice apps make you tap a screen with the hands that should be on your instrument.

We wanted music software you use like an instrument: you bring any song you love, the app turns it into something you can learn, and you steer it with your hands and voice instead of a mouse.

What it does

Overture is a music-learning website that runs on your camera and microphone. You pick a spinning record, and each record is a mode:

Learn (the core): upload any song as an MP3, even a full-band rock track. In about 10–20 seconds Overture finds its key, tempo and time signature, pulls out the melody, bass and chord parts, and writes sheet music. Note letters are there for beginners. Then you play along on a real instrument. Overture listens through the mic, measures whether you rushed or dragged, points to the exact bar ("bar 12 was early"), and gives short, friendly coaching.
- Hands-free practice: make a V sign to pause, with a 3-2-1 countdown before the music starts again. Drag the sheet to scrub, loop a single bar, or restart.
History: a progress graph of every practice run with the coaching for each one, so you can see yourself improve.
Create: hum a melody and it becomes notes. Pick an instrument, record layers over a metronome and a 4-count, then name your track and save it.
Hands as a mouse: a hand cursor follows you. Close your open palm into a fist and hold it to click, or move the fist to scroll. The live camera is the background and it reacts to the music.
Live visuals: hand positions, the beat and the music's frequency levels stream to TouchDesigner for big-screen visuals that move with you.

How we built it

Website: React + Vite + TypeScript, plus Web Audio and Tone.js:
- Playback changes tempo without changing pitch.
- MediaPipe hand tracking runs in a background worker, with our own gesture classifier on top.
- OpenSheetMusicDisplay draws the sheet music.
- A three.js shader turns the camera into the background.
Server: Python + FastAPI.
- librosa finds tempo, beats and key.
- Spotify's basic-pitch does polyphonic transcription, splitting out melody, bass and chords.
- Analysis runs in parallel worker processes, so uploads stay fast.
Gemini turns measured practice facts (timing errors per bar) into human coaching. If Gemini is slow, rule-based feedback appears instantly.
Tiger Data (Postgres + TimescaleDB) stores songs, practice runs and saved tracks. That fits well, because practice history is time-series data.
TouchDesigner gets its live data over OSC through a small relay we wrote.
How we worked: one shared repo with written data contracts between parts, a single command file anyone can edit, CI checks on every change, and automated tests for every hand and voice command, plus a browser test of the whole app.

Challenges we ran into

Full-band songs: a single-melody pitch tracker found almost nothing on rock songs. Polyphonic transcription plus splitting the result into parts fixed that. We also fixed key detection, which kept picking the wrong related key.
Telling your playing from the song: the song's own sound leaks into the mic. Practice uses headphones or a metronome, and we calibrate timing to the user's own latency.
False triggers: desk taps, key clicks and the song's drums set off sound commands. We made detection stricter while music plays and moved navigation to deliberate hand gestures you have to hold.
Performance: camera tracking, audio analysis and a WebGL background all run together in a browser. Tracking runs in a worker at 24 fps, animations use transforms only, and quality drops automatically on slower laptops.
Scope: at 6 AM we cut features (ASL letter shortcuts, a whole gesture-controlled Play mode) so that what we ship works every single time.

Accomplishments that we're proud of

On four real band songs, key and tempo came out right on all four, with separate melody, bass and chord parts in 11–15 seconds.
Feedback is specific and honest: it names the bar and whether you were early or late, instead of a vague score.
An interface you can use without touching the laptop.
Every command has an automated test that proves it actually changes the app.
A look that is fully our own: hand-drawn flowers, spinning records, and the camera as a canvas.

What we learned

Focus beats features: one clear job per screen, and an obvious signal when the app understood you.
Real-world audio is messy: testing with recorded clips from real mics and real rooms caught bugs a clean test never would.
Let AI explain the numbers, not make them up: coaching built on measured timing data is far more trustworthy than coaching built on guesses.

What's next for Overture

Pitch accuracy feedback (wrong notes), alongside the timing feedback
More instruments split out of a song, and sheet music that scrolls with you
Switching the ASL letter shortcuts and full gesture conducting back on
Practice plans built from your history ("bars 8–12 need work this week")
Multiplayer jam sessions and a phone layout

Built with

React · Vite · TypeScript · Web Audio · Tone.js · MediaPipe · OpenSheetMusicDisplay · three.js · Python · FastAPI · librosa · basic-pitch · ONNX Runtime · Gemini · Tiger Data (TimescaleDB) · TouchDesigner