Inspiration
I stutter, and I spent years in speech therapy working on it. The sessions helped, but the hardest part was always the time between them — practicing alone at home with no feedback, no idea whether a pause was a real block or just a breath, whether I was making progress or reinforcing bad habits. My therapist could hear the difference instantly; a recording of myself couldn't tell me anything.
That's the gap SpeakR is built for. Generic speech-to-text tells you what you said; none of it tells you how — where you blocked, held a sound too long, repeated a syllable, or slipped in a filler. I wanted a practice companion that pays attention to the delivery, the way a clinician does: something to use between therapy sessions that gives honest, grounded, encouraging feedback on a read-aloud take.
The second spark was a hunch about LLMs. An LLM staring at a raw transcript is blind to disfluency — but an LLM handed the right measurements might reason about them like a therapist would. I wanted to find out if that was true.
What it does
You pick a passage (or write your own), then record or upload yourself reading it. SpeakR shows:
- a metrics line (WPM, % words matched, long pauses);
- a waveform with each detected moment shaded — a block shows up as a flat silent gap, a prolongation as a sustained stretch;
- the transcript, color-coded where a disfluency was flagged;
- a table of detected moments, each with the measurement-based evidence Nemotron used;
- a personalized practice plan (the sound you struggle with + a matching tongue twister + a breathing drill);
- Coach me — the plan spoken aloud (Nemotron writes it, ElevenLabs voices it); and
- Hear it in your own voice — your voice, cloned from your recording, reading the passage fluently.
SpeakR is a practice companion, not a diagnostic or treatment tool.
How we built it
The core design principle: the pipeline measures; Nemotron interprets.
- Measure — faster-whisper transcribes with disfluency-preserving settings; difflib aligns words to the passage; Silero VAD finds voiced vs. silent regions; rule-based checks flag suspicious moments (over-long words, mid-phrase silence, repeated fragments) and compute a per-word "voiced fraction."
- Interpret — those measurements become an enriched transcript that we hand to NVIDIA Nemotron. The judge classifies each flagged moment; a planner picks one exercise and writes grounded feedback, filtered by a check that rejects any claim not supported by the measurements.
- Voice — ElevenLabs speaks the coaching debrief and does instant voice-cloning for the fluent playback.
To make sure the design actually earned its complexity, we built an evaluation harness that compares three approaches on hand-labeled clips of real stuttered speech — matching detected moments to the labels by disfluency type and timing. Everything is cached to disk so a demo survives dead Wi-Fi.
Challenges we ran into
- Whisper cleans up stutters. ASR would silently normalize "s-s-sun" into "sun," erasing the disfluency before anything downstream could see it. We fixed most of it with disfluency-preserving decoding settings — but it remains a genuine ceiling.
- A caching bug hiding in non-determinism. Whisper's word timestamps drift slightly run-to-run, which changed our judge's cache key and forced a live, ~100-second API call on every app launch. We caught it watching a "cached" clip take nearly two minutes, then fixed it by caching the transcription itself, keyed by audio content — dropping repeat loads to well under a second.
- A faster model that wouldn't stop thinking. We tried a lightweight Nemotron for speed; it ignored the "detailed thinking off" toggle and emitted unbounded reasoning that broke JSON parsing and timed out. We reverted and documented why.
- Format bugs and SDK bugs. soundfile couldn't decode the .m4a that phones and browsers produce (crashing the audio analysis), and the ElevenLabs SDK's voice-clone call was broken in our version — we routed audio through ffmpeg and cloned via the raw REST endpoint instead.
- An honest negative result. We hypothesized that feeding the judge a new acoustic feature would sharpen prolongation detection. We built it, measured it, and it produced no improvement — because a bottleneck trace showed the misses are upstream of the judge, not in classification. So we dropped it.
Accomplishments that we're proud of
- An ablation that proves the design — not just a demo. We compared three approaches head-to-head: Nemotron on the raw transcript alone, the measurements alone, and the two combined. The transcript-only LLM caught essentially nothing; the measurements alone were weak; the combination was clearly and consistently the best, catching the blocks and repetitions the others missed. Each layer earns its place, and we can show why — most hackathon projects never evaluate at all.
- Two tracks, genuinely integrated — Nemotron does the reasoning; ElevenLabs closes the loop by voicing the coaching and cloning the user's voice, not as a bolt-on.
- A bottleneck analysis, not hand-waving. We traced exactly where recall is lost and can say precisely why the classifier isn't the limiting factor.
- Engineering discipline throughout — a full test suite, a reproducible evaluation, and features that had to survive a "does this actually help?" gate before shipping.
What we learned
- Give a model the data, not the picture. An LLM reasons far better over a per-word "voiced fraction" number than over an image of a waveform. The right interface between signal processing and an LLM is numbers, not pixels.
- Measure before you ship. Two of our best decisions were deletions — a candidate acoustic feature and a faster model — both killed by the evaluation rather than by taste.
- Find the real bottleneck. Adding inputs to the judge couldn't help, because the losses happen before the judge ever sees them. Knowing where a system fails is more valuable than chasing a slightly better score.
- Privacy can be an architecture, not a feature. Running locally with no accounts and a local cache is a deliberate, defensible choice for a speech tool.
What's next for SpeakR
- A frame-level acoustic disfluency model (e.g. wav2vec2 fine-tuned on Apple's SEP-28k) to replace the rule-flag + LLM-judge classifier. It works on raw audio (immune to ASR normalization), needs no hand-tuned threshold, and can emit stacked events — removing all three bottlenecks we identified.
- Progress tracking across sessions — "you've cut blocks 30% this week" — to fully deliver the practice companion promise.
- Free-speech mode — analyze any recording without a reference passage, extending it beyond read-aloud practice.
Log in or sign up for Devpost to join the conversation.