Inspiration
Think about the last time music brought you closer to someone. Maybe it was a song everyone knew at a party, a choir at church, a friend playing guitar on a porch, or a group chat that's mostly just people sending songs back and forth. Music is one of the few things that pulls people together without anyone having to try.
But making music has a gatekeeper: money. We know people with notebooks full of lyrics and melodies they hum all day. What they don't have is a producer, studio time, or the hundreds of hours it takes to learn production software. So their songs just stay in their heads.
That felt wrong to us. The hardest part, the voice and the words, was already there. They were only missing the band.
So we built Undertone. You sing your song, and it builds the music around you.
What it does
You record yourself singing, or upload a voice memo. Undertone listens to your melody, works out the key you're singing in, and finds the beat. Then you pick a vibe (indie pop, dream pop, late-night jazz, or alternative rock) and how energetic you want it, and it creates chords, keys, bass, and drums that follow your timing.
From there you can make it yours: gently tune your vocal, turn tracks up or down, mute or solo them, try new chords, and export the finished song.
It all runs on your own computer. Your unreleased song never gets uploaded anywhere, because it's yours.
How we built it
The app runs in the browser, with a small Python server on your own machine doing the heavy audio work.
To understand your voice, we use AI pitch models (SwiftF0, RMVPE, and Spotify's Basic Pitch) that trace the melody you sang note by note. A beat-tracking model called Beat This! listens for the pulse of your song. For tuning, we use a tool called WORLD that can shift pitch while keeping your voice sounding like you.
The band itself is generated in code. We pick chords that match what you sang and flow naturally from one bar to the next, then play them on synthesized piano, strings, bass, and drums.
One idea we're proud of is how gentle the pitch correction is. Instead of snapping every note into place, we measure how far off a note is in cents (hundredths of a semitone):
$$ \text{cents off} = 1200 \log_2\left(\frac{\text{your pitch}}{\text{target pitch}}\right) $$
We only nudge notes that are a little off. If a note is way off (more than 65 cents), we assume you meant it, like a slide or a bit of style, and leave it alone. That keeps the vibrato and character in your voice.
Challenges we ran into
Finding the beat with no drums. Beat-tracking AI is trained on full songs with drums and bass. A single voice in a bedroom is much harder to read. The AI sometimes guessed the wrong tempo or started counting in the wrong place. Instead of pretending it was perfect, we gave artists simple ways to fix it: tap the tempo, drag the beat markers, or say "this is where the song starts."
Making tuning sound human. Our first attempts made people sound like robots. It took a lot of listening and adjusting before corrected vocals sounded like a better take of the same person, not a different person.
It didn't work on everyone's laptop. One of the AI models kept crashing on some computers because of graphics-card issues. Since the whole point is that anyone can use this, we had to rework how it loads so it runs on ordinary machines.
No "right answer" to test against. We didn't have a dataset of real community singers with perfect labels, so we couldn't honestly say which AI model was best. We built a way to compare them side by side and listen to what each one heard.
What we learned
Our biggest lesson was that the AI shouldn't have the final say. The best results came when the AI did the tedious work and the artist made the creative calls. We also learned a lot about how sound works: how pitch is measured, why a voice sounds like your voice, and how much small timing details matter when you're trying to make something feel like music.
And we learned to be honest about what our tool can and can't do. Some voices and songs will still trip it up, and we'd rather say so than overpromise.
What's next
We want to put Undertone in front of real people: local artists, youth music programs, church choirs, and anyone with a song they've never been able to hear out loud. We'd love to add more genres, real recorded instruments, and a way to share songs and collaborate, so one person's voice memo could become a whole community's song.
