Inspiration

Imagine waking up in an intensive care unit. You are fully conscious, surrounded by medical equipment, and desperately need to tell someone that you're in pain. You try to speak, but no sound comes out.

For many patients receiving invasive mechanical ventilation, this is a frightening reality. Although intubation can be lifesaving, the endotracheal tube prevents normal vocalization, leaving conscious patients dependent on gestures, lip movements, and assistive communication devices.

The consequences extend beyond frustration. A 2022 study found that 82.2% of communication exchanges about pain were unsuccessful among nonspeaking ICU patients. When even basic needs cannot be communicated reliably, patients can experience fear, helplessness, stress that may worsen their condition or even loss of control.

Existing solutions, including communication boards and eye-tracking devices, have shown promise. However, some patients may struggle with fatigue, limited movement, or the effort required to interact with these systems.

We created Verba to explore a simpler way for conscious, voiceless patients to express their needs and reconnect with the people around them.

What it does

Verba translates subtle, intentional movements into audible speech, providing an alternative communication channel for conscious ICU patients who cannot vocalize.

Rather than requiring patients to type or speak, the system is designed to recognize a small vocabulary of deliberate gestures and associate them with essential messages.

A patient could communicate: "Yes" “No” "I'm in pain" "I can’t" "I’m thirsty" “I’m scared” “Where’s my mum” “Thank you” “Where’s my family?” “I need suction” “I’m cold” “I love you”

The recognized message is converted into speech through AI-powered voice synthesis. Where an authorized voice sample is available, personalized voice generation could help patients communicate in a familiar voice - one that effectively mimics their own. Verba is designed to support rapid, accessible communication while keeping the patient's intended message at the center of the interaction.

How we built it

Verba is a phone app that lets ICU patients on a breathing tube speak again. They silently mouth a phrase, the phone reads their lips, and it speaks the phrase in their own cloned voice. All of the lip reading happens on the phone itself. No video ever leaves the device.

MediaPipe Face Landmarker runs in the browser and tracks 40 lip points plus 4 reference points on every frame. We normalise those points for head position, tilt and distance. Each mouthed phrase then becomes a 3,840-number motion signature, which a small neural network (an MLP) turns into a guess. Training in Python, running on the phone. We recorded teammates and trained the model in Python. We store lip landmarks only, never video. The model is exported to JSON, and an exact TypeScript port runs it on the phone. An automated test checks the phone’s maths against Python’s, and they match to about 15 decimal places.

We calibrated the model with temperature scaling, so “86% sure” really means it’s right about 86% of the time. If it’s 70% sure or more, it speaks. Below that, it says it isn’t sure and shows its top guesses. The patient then holds a long blink to confirm or blinks twice to move to the next guess. We used Cartesia Instant Voice Cloning to clone a patient’s voice from 10–60 seconds of an old family video, and Sonic 3.6 to pre-generate every phrase. That makes speaking instant and lets it work offline at the bedside. A small server function also lets a family clone a voice live from inside the app, and the API key never reaches the phone.

“Teach it their lips.” In about 2 minutes, the patient mouths each phrase 3 times. The phone then fine-tunes the model’s last layer to that person, entirely on the device. When a family member asks “Are you in pain?”, the app re-orders its guesses so likely answers rise to the top. A question can never make the app speak on its own. It’s built in React, TypeScript, Vite and Tailwind as an installable PWA. It has patient profiles, a talk screen, a phrase board as a fallback, undo, and a log of every message. It deploys automatically to Vercel through GitHub Actions.

Challenges we ran into

Reading lips of people the model has never seen. Our first models got only 38–48% right on the first guess for a new person. Switching from logistic regression to an MLP, adding data augmentation and error analysis brought that to 53% first guess and 81% top 3. That still isn’t good enough to speak blindly, so we designed the whole experience around it: blink-to-confirm, a ranked phrase board, and calibrated confidence.

Making the phone and Python agree.The on-device feature pipeline had to match Python exactly, or the model would quietly break. We built a “golden” test fixture to catch any drift.Phones are strict. iPhones only allow camera access over HTTPS, and they block audio until the user taps something. We had to unlock audio on the Listen tap so the cloned voice could play.

Deciding when someone has finished mouthing. We tuned the app to stop capturing automatically once the mouth has moved and then stayed still for 0.9 seconds. Blink detection had to tell a deliberate long blink apart from a natural one.

Deploys. Free plan blocked deploys of commits from teammates, so we moved to deploying from GitHub Actions on every push. Pivoting mid-hackathon. We started with a bedside screen and a laptop. We pivoted to a single phone app with on-device lip reading, because that’s what a family member would actually have in their hand.

Accomplishments that we're proud of

Lip reading runs entirely on a phone, with no server and no video leaving the device, and it works on bad Wi-Fi. Personal calibration works. 3 takes per phrase raised accuracy from 53% to 62% first guess and from 81% to 87% top 3. It improved for every person we held out. The share of phrases the app could speak straight away, without confirming, rose from 12% to 33%, and those were right 84% of the time. The demo: a teammate with a scarf over their throat silently mouths “Where’s my mom?”, and the room hears it in their own voice. We’re honest about uncertainty. The app never says anything in the patient’s voice without their yes, every message can be undone, and everything is logged. A calm, accessible design for a scared family member in a dim room: large type, one decision at a time, no red, and good contrast in light and dark mode.

What we learned

With a small dataset, being honest about uncertainty matters more than raw accuracy. A well-calibrated “I’m not sure, did you mean this?” is safer and more useful than a confident wrong answer. Testing on people the model has never seen gives the real number. It was humbling, but it shaped every design decision. A few minutes of personal data goes a long way. Fine-tuning just the last layer was fast, private and effective. Voice is identity. Hearing a patient’s own voice instead of a robotic one changes the whole experience, and it requires real consent. Clear ownership and shared contracts let three people build quickly without stepping on each other’s work.

What's next for Verba

Testing with clinicians and real patients. That means ICU conditions: tubes and tape around the mouth, patients lying flat, low light. It also means feedback from the nurses and respiratory therapists we spoke to. More phrases. We have 50 phrases written and voiced, but the shipped model only knows 8. More training data from more people will unlock the rest. More training data recorded on phone cameras, at the angle a family member would actually hold the phone. Support for more languages and for patients who prefer their native language. A path toward clinical validation, with privacy and consent built in from the start.