Inspiration:
Talking with the representatives at ALS Canada really opened our eyes to the painful realities that people suffering with this disease face. Hearing them speak about how patients feel like they lose their connections and become ostracized from their communities was devastating. They mentioned how difficult it is for those with ALS to speak to their loved ones, and express their feelings. Current speaking alternatives are slow, expensive, and most importantly, they take the human connection out of conversation. We wanted to engineer a way to replace staring at a screen for five minutes just to say “I love you”, to simply mouthing the words and hearing your voice say it. That’s why we built LouVoice.
What it does:
By using the sensors already built in to a standard pair of AirPods, LouVoice tracks the subtle facial muscle movements you make and trains a machine learning model to recognize the specific movements of each word. Once you calibrate the model by performing whatever mouth movement you are able to for a specific word, you can simply mouth a sentence and the model will recognize it and display it on the computer, but here’s the coolest part: using ElevenLabs’ ultra-realistic AI voice clones, as soon as you mouth your sentence, it is read aloud, in your voice!
How we built it:
We split the work by strength. Konrad built the app, the recognition model and the website. Neel led product development: he wrote the PRD, configured ElevenLabs, Vultr and Tiger Data, and spoke directly with ALS Canada and doctor Suraj Tandon to gather the insights that shaped what we built.
LouVoice is a native macOS app written in Swift. AirPods Pro 2 stream head-motion data to the Mac, and when you mouth a word, your jaw moves the earbuds just enough for the motion sensors to pick it up. No camera and no microphone are used to recognize words. Reading the motion: every capture is corrected for how the AirPods are sitting in your ears, then filtered down to the movement that comes from the jaw.
Recognizing words: A small neural network trained in PyTorch on our own recordings picks the word. It runs as about 100 lines of plain Swift and takes 15–20 ms per word. A dedicated “not a word” class catches chewing, swallowing, nods and head turns, so stray movement doesn’t turn into a guess. Speaking it: The word goes to ElevenLabs and is spoken in the user’s own cloned voice. The audio is cached on the Mac, so repeated words play instantly, even offline, and the API key stays in the macOS Keychain.
Hands-free control: Nod to accept a word, shake your head to reject it. Tracking change over time: Every attempt is stored in a Tiger Data (TimescaleDB) hypertable. A continuous aggregate turns it into a weekly progress curve, so caregivers can see when a word starts slipping.
The website: louvoice.health (a GoDaddy Registry .health domain) runs on a Vultr server and pulls that progress curve live from Tiger Data. How we coded: We built with Claude Code, and Entire recorded every session as a checkpoint next to our commits (64 this weekend).
Challenges we ran into:
The biggest issue we faced was calibrating the model. In our first sessions it scored 81% on recordings it had never seen, but the moment the AirPods sat even slightly differently in our ears, live accuracy collapsed to 38%. Every session, every re-seat and every tired late-night take looked a little different to the model. So we tested and iterated over and over. Overnight, we ran an experiment loop that tried about 80 ideas and 191 model configurations over roughly eight hours, keeping only what actually improved accuracy on recordings the model had never seen. We also added a quick calibration step: before each session you mouth each word four times (about 80 seconds), and the model adapts to that day’s fit in about two seconds. That brought live accuracy back from 38% to 64%, and the overnight run’s improvements went into the version we demo.
Accomplishments that we’re proud of
45 out of 45: demo mode chose the right sentence every time, between our two trained sentences. 754 real AirPods recordings made by our team this weekend. Dr. Suraj Tandon (palliative care and internal medicine) called LouVoice “very, very viable,” and the ALS Canada representatives called it “absolutely fantastic" and "incredibly thoughtful.” A full stack built in one weekend: the Mac app, the voice clone, the database, a live website and a waitlist.
What we learned:
Just how much room there is for innovation and improvement in this field. After speaking to Dr. Tandon and the ALS Canada representatives, we now see just how important it is to raise awareness for this disease. Often times disease like cancer get all of the publicity, but the people who struggle with this terminal disease deserve to have as much support as they can. We hope to use LouVoice to help these patients keep their connections strong despite their illness and maintain their dignity.
What’s next for LouVoice
Train every user’s model on Vultr Cloud GPUs. Retraining the whole network instead of only the last layer added about 7 more points in our tests, and that needs real GPU training. The new model then comes back to the Mac so recognition still works offline. Adapt as ALS progresses. Use the Tiger Data history to spot when a word starts slipping and ask the user to retrain just that word, and tell users and caregivers honestly when the signal gets too weak to read. Test with people living with ALS, starting with the clinic connections ALS Canada offered across Canada. Beyond ALS: Dr. Tandon pointed out that non-verbal patients are some of the hardest to treat in the ER and ICU. Work in reverse: turn mouthed words into typed messages for people who can’t use their hands.
Built With
- caddy
- coremotion
- elevenlabs
- entire
- godaddy-registry
- node.js
- numpy
- pm2
- postgresql
- python
- pytorch
- scikit-learn
- scipy
- shadcn-ui
- signal-processing
- swift
- tiger-data
- timescaledb
- vultr