Patient Listener

Inspiration

Most voice assistants are built for people who speak quickly and clearly. For someone recovering from a stroke, living with a stammer, or managing Parkinson's or MND, those tools do the one thing that hurts most: they jump in, finish the sentence, and move on. We wanted to build the opposite. A partner that waits as long as it takes, never guesses ahead, and treats slow speech as normal rather than as an error to correct.

What it does

Patient Listener is a voice practice partner for adults whose speech takes time. It listens for the words a person is actually working on, waits for them to finish on their own, and gives quiet encouragement when a word lands. The practice list is personal: a carer or speech therapist chooses the words that matter to that individual, so practice stays relevant to daily life rather than a generic script.

How we built it

The system is built around a deterministic decision core, with the models kept deliberately at the edges. The core rule, verbatim from our engineering workbook:

Code decides when to speak and what counts as a found word; models only choose the wording of a hint or reply, from facts the code supplies; a person (carer or SLT) owns the word list.

Timing, turn-taking, and word matching all live in plain code we can read and test. The models sit around that core, each with a narrow job:

• Live speech recognition runs on a local NVIDIA GPU through NeMo, using nvidia/nemotron-speech-streaming-en-0.6b. Keeping recognition local means low latency and no round trip while someone is mid-word.
• Spoken prompts, replies, and prepared hints are voiced on the same local GPU through NeMo with nvidia/magpie_tts_multilingual_357m.
• Short replies and profile-specific completeness checks go to nvidia/Nemotron-3_5-Lightning on Nebius Token Factory, a small fast model for the quick in-session decisions.
• AI-assisted meaning hints during plan generation use the larger nvidia/nemotron-3-super-120b-a12b, also on Nebius Token Factory, where a bit more reasoning helps and latency matters less.

The repository is the implementation of that decision core plus the ports and adapters around it, so recognition, speech, and the two Nebius models can each be swapped without touching the logic that keeps the experience safe.

Challenges we ran into

The hardest problem was timing. Deciding when someone has finished a word rather than just paused mid-attempt is not something we wanted a model guessing at, so we worked out thresholds in code and tuned them against real patterns of slow speech. Splitting the work across local and hosted models was the other challenge: the streaming recogniser and TTS had to stay on the local GPU to keep latency down during a turn, while the Nebius-hosted models handle the steps where a short wait is acceptable. Getting that division right, and keeping each model strictly inside its lane, took most of the design effort.

Accomplishments that we're proud of

We kept every judgement that affects a person's dignity out of the models and in code we can inspect. Recognition and speech run locally for responsiveness, the heavier reasoning runs on Nebius Token Factory, and the architecture is clean enough that the word list genuinely belongs to a carer or therapist, not to us and not to a model. And the core behaviour, waiting without interrupting, works the way we set out for it to work.

What we learned

The biggest lesson was where to put the AI and where to keep it out. For a care-facing tool, the trustworthy parts are the deterministic ones, and a model earns its place only in a narrow task. We also learned a lot about matching the model to the job: a small model like Nemotron-3.5-Lightning is the right tool for fast in-session checks, while the 120b model is worth its latency only at plan-generation time. Designing with those splits from the start, rather than wiring one model into everything, made the whole system simpler to reason about.

What's next for Patient Listener

We want to get it in front of real speech therapists and the people they work with, and let that feedback shape the word lists and the pacing. The multilingual TTS already opens a path to more languages, so we would like to extend recognition and practice beyond English, add progress tracking that a carer or clinician can actually use, and keep the same discipline about what each model is and isn't allowed to decide.

Built With

• ai
• nebius
• python