OpenSound

The ML-powered learning tool for Deaf and hard-of-hearing string musicians that provides real-time haptic and visual feedback while they play.

Comment

What did you build, and how does it address the prompt?

We're giving people with hearing impairments a way to feel and play music :)

Some of our siblings, friends and floormates are Deaf or hard of hearing. They love music as much as we do, and most music tools quietly assume you can hear.

The violin is the hardest case. It has no frets, so every note is placed and corrected by ear, to within a few hundredths of a semitone. A tuner app helps... until your eyes are on the sheet music and your hands are on the bow.

The prompt asked us to take something people struggle to use and widen who can use it. For us that's practising an instrument alone.

OpenSound moves tuning feedback from your ears to your body.

You play a note. The laptop hears it and compares it with the note you were meant to play, then answers by touch:

You feel... It means... So you...
A thump on your left You're flat Go up
A thump on your right You're sharp Go down
A quick thump on both sides You're in tune Keep playing
One thump A little off Small nudge
Two thumps Far off Big move

The side tells you which way to move. The rhythm tells you how far. So you never have to look away from the music.

Around that cue is a place to learn on your own

  • Pick a song and it waits for you, note by note.
  • Play along at tempo, then review each mistake with the camera's picture of your hand.
  • Bring your own music by pasting a link, typing notes or dropping in a file, and it becomes a lesson.
  • The camera finds the violin's fingerboard and lights up where your finger goes next.

Three things we cared about most

1 · Your practice stays yours

Learning an instrument means sounding bad for a long time, and nobody should have to send that, or a video of their face, to a server.

Pitch detection, hand tracking, fingerboard finding and face reading all run in the browser, on your own laptop.

  • The audio, the video and your face/fingerprints never leave the computer.
  • Nothing is saved.
  • You don't need an account.

2 · Calibrated so it never overwhelms

Touch is personal. What feels like a gentle tap to one person is too much for another.

Instead of choosing one fixed intensity and hoping it works for everyone, OpenSound continuously adapts the haptic feedback while you play.

Our calibration algorithm uses each speaker's volume, measured latency and microphone feedback to keep adjusting how the vest responds. It tries to make every cue arrive at the right time and at a comfortable, consistent intensity, even when the speakers themselves behave differently.

Feature What it does
Per-speaker levels Every speaker has its own independently controlled level, so the system can compensate when one side is naturally louder or quieter than another.
Measure Sends test thumps through the system, listens for them through the microphone, calculates each Bluetooth speaker's delay, and automatically compensates for that latency. That way "in tune" feels like one clean tap instead of several speakers firing at slightly different times.
60-second leak check Measures how much of the vest's own vibration and audio reaches the microphone, helping the algorithm avoid confusing its own feedback with the violin.
Live calibration The system keeps using those calibration values while you practise, instead of treating calibration as a one-time setting.
Play-only response The vest only answers when you play. Silence means silence.

The goal: the player thinks about the music, not about managing the vest.

3 · It checks that you're enjoying it

We didn't want to build a drill sergeant.

Alongside the music models, we built our own lightweight, on-device expression model to give the player a small signal about how practice is going.

We trained and tested it using a mix of:

  • Public facial-expression datasets
  • Examples collected from ourselves and friends while actually playing instruments

That gave us examples closer to what the model would see in practice: people concentrating, smiling, struggling with a passage or simply staring intensely at sheet music.

The model runs locally and classifies the camera view into simple states:

Smiling  ·  Neutral  ·  Tense

If the player has looked tense for a while, OpenSound can suggest taking a break, playing something they enjoy or stopping for the day.

The point is to share the love of music, not to grind someone through scales.

We deliberately treat this as a hint, never a score. A facial expression is not the same thing as an emotion, and that distinction matters even more for Deaf users, because facial expressions can also carry linguistic meaning in sign language. Like the rest of OpenSound's vision processing, the model runs on the device, and the camera data never needs to leave the laptop.

How did you build it?

Hearing the note

The room page runs in Chrome on a Mac. It reads the microphone with the Web Audio API and detects pitch with the McLeod method (pitchy), keeping only clear notes between 70 and 1500 Hz.

Measuring how far off you are

A small Python server measures how far off you are in cents:

$$c = 1200 \log_2 \left( \frac{f_{\text{played}}}{f_{\text{target}}} \right)$$

Picking the cue

Cents off Cue
≤ 15 (held for 150 ms) In tune
16 – 24 A little flat / sharp
≥ 25 Far flat / sharp

One source of truth. The cue rules live in one Python module, and the page, the iPhone app and the vest all read from it, so they can't disagree.

Each Bluetooth speaker gets its own audio stream, so left thumps reach only the left side.

The rest of the stack

Piece Built with
iPhone app Written in Swift. Shows the cue and taps it with the phone's own haptics.
Camera MediaPipe for hands and faces, plus a fingerboard model running in ONNX in the browser.

What fought back

Problem What happened
Bluetooth doesn't scale. A laptop holds about three speakers before audio stutters, and an iPhone streams to only one, so we split the vest between the Mac and a phone.
The vest heard itself. A thump near the mic was read as a note and produced a false "flat". That's why the leak check exists.
Small speakers can't play deep bass. Our first thumps were too low to reproduce, so we raised the pitch.
Every speaker lags differently. That's why Measure exists.

The business side

The market

The WHO estimates about 430 million people live with disabling hearing loss. We couldn't find a single practice tool built around feeling pitch instead of hearing it.

Right now, if you're Deaf or hard of hearing and want to learn an instrument, your real option is a private tutor. That's expensive, hard to find, and only helps for the hour they're in the room.

Private tutor Tuner app OpenSound
Cost Expensive Cheap Free, open-source
Works without hearing Depends on the tutor Only if you're watching it Yes, by touch
Eyes stay on the music Yes No Yes
Practise alone, any time No Yes Yes

OpenSound is the first tool built for this. Not an adapted hearing tool. Built for it.

Traction so far

  • Met with the UIUC Department of Music about bringing OpenSound to real students and teachers.
  • Posted on Reddit and got overwhelmingly positive feedback from Deaf and hard-of-hearing musicians and the people who play with them.

How we scale: the Khan Academy playbook

Khan Academy took something that used to need a tutor and made it free for anyone with an internet connection. We want to do the same for music, and be as big as Khan Academy for people who feel music instead of hearing it.

Step What it looks like
1 · The phone The iPhone app already taps flat, in tune and sharp with three different patterns on its own haptic motor. No extra hardware. One of us has tried it blind.
2 · Open source Teachers record lessons, players share arrangements, and anyone can turn a song they love into a lesson for someone else.
3 · A place to share Like SoundCloud, but built from the start for players who feel music. Post and share what you play, not just practise.
4 · The belt Our quiet two-motor belt for players who want left and right on the body without the noise.

The vest you see here is a proof of concept, not the product. We built it from speakers we already owned and a jacket from Target, and nothing was soldered. It exists to prove one idea.

Built With

Share this project:

Updates

Submission history