Silent Voice

Cancer can take your voice, but we can give it back.

Comment

Inspiration

Imagine you go into surgery able to talk, and when you wake up you try to say your kid's name and nothing comes out, not even a whisper. That's a laryngectomy, the surgery that removes the voice box, usually to save someone from throat cancer. Every year close to 190,000 people around the world find out they have cancer of the voice box, and more than half a million are already living without one.

But did you know losing your voice box doesn't take your words with it. Your lips, tongue, and jaw still move exactly the way they always did, so you can still mouth "I love you" or "I'm scared" or a bad joke at dinner, but nobody hears it. The tools people get today, like a buzzing device pressed against the neck, give back a sound, but almost never the voice their family knows.

The loving words are all still there. Only the sound is missing.

That felt like something we could fix. If a machine could read those movements fast enough, it could give people their own voice back in real time as they speak.

What it does

That's what inspired us to build Silent Voice. We want to help people reclaim their voice, so they can talk to the people they love again, face to face or even over the phone.

The people who feel it most are their families. A mom who hasn't heard her son since his surgery picks up the phone, and it's him, in his own voice, telling her about his day. Or a grandfather reads a bedtime story again, and his grandkids hear the same voice they grew up with.

Now their families get to hear them again with Silent Voice. Really hear them, again!!

Silent Voice turns silent mouthing into speech in your own voice, while you're still mouthing. Our demo is a live phone call. Austin, our teammate, calls you on an ordinary phone line, so there's nothing for you to install. You talk normally, and your words show up in his live captions. He answers by mouthing, mouth open and no sound at all, and you hear him in his own cloned voice, carrying the emotion on his face, about 0.7 seconds after his lips start each word. Each word plays the moment the system is sure of it, so he never waits for the sentence to end. When it can't tell what he said, it says "Please repeat" into the call instead of guessing, because putting the wrong words in someone's mouth is worse than a pause.

For the person on the other end, it sounds just like Austin, just a beat slower.

How we built it

Any singular method here is unreliable on its own. For example, if you say the words "bat", "mat" and "pat", they look identical on camera, and it can't tell which is which. So Silent Voice has three readers, each covering what the others miss, combining them into the best possible answer live.

Lips. Auto-AVSR, an open-source lip reader, reads finished video clips. We re-read the sentence every 100 ms, with a lightweight add-on trained on the user's face.

Muscles. Four sensor pads on the face and under the chin pick up the electrical signals of the muscles that move the lips and tongue, which the camera can't see. They feed an Arduino at 1,000 samples a second. Our muscle model, SVR-1, learned from Berkeley's silent-speech recordings, before being further fine-tuned on Austin's recordings.

Conversation. Claude hears both sides of the call. Claude Haiku predicts Austin's next words before he mouths them, and Claude Opus fixes look-alike mix-ups in the captions as they happen. Our code only accepts a swap of one word for a look-alike the readers already considered, so Claude can never put words in Austin's mouth. We also trained our own language model on 72 million words of real dialogue, and it predicts conversation three times better than the audiobook model used in previous research, at a ninth of the size.

Voice. We cloned Austin's voice with Cartesia and streamed it to Sonic 3.6, using both his expression and his facial muscles to portray his emotions. A smile will make his voice happier, and clenching his jaw will make the voice louder. Sonic outputs phone audio straight into Twilio, and Cartesia Ink transcribes what you say from the same call.

Everything runs on one laptop (RTX 5060), with every message logged and replayable. We built it with Claude Code.

Challenges we ran into

Making Auto-AVSR read silent speech live

Auto-AVSR is one of the best open-source lip readers, but it was never built to be live. It looks through the whole video before answering, which makes it completely unusable for us. Every other real-time lip reader gives up accuracy for speed instead, like LiteVSR, which has an error rate of 46% on its own benchmark. We needed both, so we optimized it wherever possible. To make it live, every 100 ms we reread the whole sentence so far with a fast greedy pass that reuses the work on frames it has already seen, so each pass takes under 40 ms. To improve speed and accuracy, we dynamically crop the footage to only the user's mouth and retrained the lip tracker on Austin's face. In the end, on a brand new test session, our error rate fell to 9.8%, down from Auto-AVSR's 20.9% on the same test.

Making Claude think ahead

The words "bills" and "pills" look identical to a camera, so you need an LLM to follow the conversation and correct it. The difference between us and other models is that we stream every word live and can't wait 7 seconds for Claude like they can. So, we turned Claude into a forecaster instead of an editor. While the caller is still talking, Cartesia Ink streams their half-finished sentence to Claude, and with two predictions always in flight, a fresh guess at Austin's reply lands perfectly on time, reaching our decoder before Austin's lips even move, tipping close calls toward words that fit the conversation.

Making word-by-word speech sound human

Cartesia needs to know where the sentence is going to properly add emotion, but our lip reader is live and hands over one word at a time to read immediately. With Sonic's buffer on, it held 14 of 15 words until the sentence ended, and the caller sat through 4.9 seconds of silence. With words sent one by one, it treated each word like a sentence, with a heavy pause between them. So, we taught Silent Voice to phrase like a person. It waits for Austin's natural pauses and sends a few words at a time into one continuing stream, so Sonic carries the melody from phrase to phrase. To keep it fast, we trim Sonic's padding as the audio arrives. The caller hears him about 0.7 seconds after he starts a word, making it as seamless as possible.

Accomplishments that we're proud of

  • A live phone call where one side only mouths, and you hear it as speech while it's being mouthed.
  • Speech starting about 0.7 s after the lips do, in the speaker's own voice, with the emotion on his face.
  • A lip reader that answers every 100 ms and got 90.2% of words right on sentences it had never seen, up from 79.1% for Auto-AVSR out of the box.
  • A muscle model built for just four pads. Given the same four, it picks the right sentence among 10 lip look-alikes 72.3% of the time, against 27.7% for the published model.
  • Claude predicting Austin's reply while the caller is still talking, with a fresh guess about every 0.8 s.
  • Building this all in only 36 hours.

What we learned

We expected Austin's lip-reading mistakes to be look-alikes like "bills" and "pills". When we went through them one by one, most were short words he'd dropped and contractions like "I'd" read as "I", so that's where our last day went.

We switched Claude to Haiku for speed and replayed a test session through it. It changed 31 words, and most of them were right words turned wrong: "sleep" became "slip", "think" became "thank", "creative" became "kind". Opus changed 3 on the same session. So Opus fixes the captions now, and Claude on its own can never put a word in Austin's voice.

The emotion reader caught us off guard too. Trained on one session, it called every neutral sentence in the next one emotional. It had learned Austin's resting face from his first three seconds, and he'd started that session with his eyebrows raised. Now it compares him to the last minute of his own face.

What's next for Silent Voice

  • Testing with laryngectomees and speech therapists.
  • Measuring impact in real calls: how often the listener understands on the first try compared with an electrolarynx, how often the system says "Please repeat", and how fast each word reaches the listener.
  • A shorter setup for new users. The lip add-on trains in minutes; the muscle model needs a few recording sessions.
  • Barge-in, so the voice stops when the other person talks over it.
  • Hardware small enough to wear to a coffee shop.

Built with

python, pytorch, auto-avsr, mediapipe, hsemotion, kenlm, claude-api, claude-code, cartesia-sonic, cartesia-ink, twilio, zeromq, arduino, bioamp-exg-pill

Team

Thai (EMG rig, project manager), Austin (SVR-1, combining the readers, demo speaker), Frank (Cartesia voice, Twilio call, dashboard), Evan (real-time lip reading/auto avsr, Claude predictor).

Built on Gaddy and Klein, Digital Voicing of Silent Speech (2020), Auto-AVSR (Ma et al., 2023) and Lipflow.

Built With

Share this project:

Updates

Submission history