Inspiration
Most speech-feedback tools give you a number ("confidence: 72%") or a vague comment ("try to vary your tone"), and neither tells you where in your speech something went wrong or why the tool thinks so. We wanted feedback you could check: point to the exact second, show the measurement, name the rule, and let the speaker hear it for themselves.
Track C's framing made this concrete. If you have a good reference reading of a passage and a second reading of the same text, the reference becomes a ruler, and every deviation can be measured against it instead of against some abstract idea of "good speech".
What it does
SpeechLens compares a participant's recording of a passage against a reference recording of the same text and pinpoints delivery flaws. Every flaw comes with:
- a start and end timestamp, and the word it follows or spans,
- the measurements from both recordings,
- the rule and threshold that triggered it,
- the observed evidence, kept separate from the inferred interpretation, plus a recommendation.
For example: "Participant pause of 2.99 s after the word 'develop'. Reference pause at the same position: 1.01 s." That is the evidence. "This may interrupt the flow between ideas" is the interpretation, and it is labelled as one.
It currently detects overlong pauses, local and overall pacing changes, and flat pitch. A deterministic rubric scores pacing, pause control and vocal variation, and the dashboard shows both waveforms with flaw regions, a clickable word-level transcript, and click-to-seek on every flaw.
There is no LLM anywhere in detection or scoring. Every number comes from the audio, every threshold lives in a config file, and anything that can't be measured is shown as unavailable, with the reason, never guessed.
How we built it
Pipeline: decode → features → forced alignment → word-indexed comparison → rules → rubric → dashboard.
Features are computed on a shared 10 ms grid with librosa: RMS energy (25 ms window), voice activity and pauses, and pitch with pYIN (64 ms window, 65–400 Hz). Two choices make comparisons fair across speakers and microphones:
- The silence threshold is relative to each recording's own level, so the same speech at half the volume gives identical pauses: $$\text{speech}(t) \iff 20\log_{10}\frac{\mathrm{RMS}(t)}{\operatorname{median}_\tau \mathrm{RMS}(\tau)} > -20\ \text{dB}$$
- Pitch variation is measured in semitones relative to each speaker's own median, so a higher or lower voice isn't mistaken for more or less expressiveness: $$s(t) = 12\log_2\frac{f_0(t)}{\operatorname{median}(f_0)}, \qquad \text{flat if } \frac{\mathrm{IQR}(s_{\text{part}})}{\mathrm{IQR}(s_{\text{ref}})} < 0.5$$
Forced alignment. We wrote our own CTC forced aligner: wav2vec2-base-960h produces per-frame character log-probabilities, and a Viterbi trellis implemented in NumPy finds the best path through the blank-interleaved transcript:
$$\delta_t(j) = \log p_t(y_j) + \max\big(\delta_{t-1}(j),\ \delta_{t-1}(j-1),\ \delta_{t-1}(j-2)\big)$$
where the $j-2$ transition is allowed only between distinct non-blank characters. This gives every word a start, end and alignment score at 20 ms resolution. It runs in under 1 GB of VRAM on a 4 GB laptop GPU, and takes about 8 s for a 3-minute recording once the model is loaded. If PyTorch isn't installed, the app falls back to an energy-only mode and says so on screen.
Comparison. Two readings of the same text have different lengths, so comparing "the same second" in each would compare different words. We match words by transcript position, never by time, and we deliberately don't time-warp the recordings onto each other, because warping would erase the pacing differences we're trying to detect. A pause is assigned to the word gap it falls in, so the participant's pause after word $i$ is compared with the reference's pause after word $i$. Local pacing compares onset-to-onset spans over a sliding window of 5 words: $$r_k = \frac{t^{\text{part}}{k+5} - t^{\text{part}}{k}}{t^{\text{ref}}{k+5} - t^{\text{ref}}{k}}, \qquad \text{flagged if } r_k \notin [0.6,\ 1.4]$$
Scoring is a weighted mean of the available dimensions; if one can't be measured, it's dropped and the weights renormalised: $$\text{Overall} = \frac{\sum_{d \in A} w_d S_d}{\sum_{d \in A} w_d}$$
Stack: Python, FastAPI, librosa, PyAV, PyTorch and transformers on the backend; React, TypeScript, Tailwind and WaveSurfer.js on the frontend. Everything runs locally with no paid APIs, and the core pipeline is covered by automated tests built on signals with known properties: tones of known frequency, silences at known times, gain-shifted copies.
Challenges we ran into
- Pitch tracking that silently failed. Our first version marked only 2% of speech frames as voiced. The tests still passed, because pure tones were tracked perfectly. Tracing it stage by stage showed a confidence threshold was discarding 97% of the frames pYIN had correctly called voiced: its "probability" output isn't a calibrated confidence. Removing the threshold brought voicing to a credible 55–77%.
- Default windows built for music. librosa's defaults gave 128 ms analysis windows, which smeared pause boundaries by up to 100 ms. Switching to speech-appropriate 25 ms windows with a 10 ms hop brought boundaries within 25 ms.
- Off-by-a-frame time axes. Pitch and energy disagreed by two frames because of an inconsistent frame-count formula, which would have misaligned our plots. We fixed it and added a test asserting both time axes are identical.
- Word ends are less reliable than word starts. Measured against detected pauses, aligned word starts landed within about 30 ms, but word ends came 20–190 ms early. So pause boundaries come from the energy detector, and local pacing uses onset-to-onset intervals instead of word durations.
- Resisting the urge to loosen tests until they pass. Twice a failing test was "fixed" by widening its tolerance until it could no longer catch the bug it was written for. We made it a rule to fix the cause, not the tolerance.
Accomplishments that we're proud of
- A from-scratch CTC forced aligner in NumPy, unit-tested on hand-built emission matrices with known best paths.
- Every flaw is traceable: timestamp, word, both measurements, rule, threshold. A judge can click a flaw and hear exactly what triggered it.
- Honest reporting: no invented confidence scores and no guessed values. When the system can't measure something, it says so and explains why.
- Speaker- and gain-invariant measurements, backed by tests that prove the invariance.
What we learned
- Passing tests aren't the same as correct measurements. Synthetic signals hid the pitch bug; only checking intermediate numbers stage by stage exposed it.
- Signal-processing defaults encode assumptions about the signal. Music defaults don't fit speech.
- In comparative analysis, what you align on matters as much as how: aligning by words instead of by time is what made the comparisons meaningful.
- Separating evidence from interpretation keeps a tool honest. Low pitch variation is a measurement; "lacks confidence" is a claim the audio alone can't support.
What's next for SpeechLens
- Calibrate thresholds on human recordings. Current thresholds are provisional, set from first principles and marked as such.
- Grow the contrastive dataset of the same passages read cleanly and with controlled flaws, with labelled flaw boundaries, and report precision, recall and temporal IoU on it.
- De-duplicate overlapping detections, since a long pause currently also triggers a local-pacing flaw.
- Synchronised reference-vs-participant plots of pitch and energy, and a dataset browser.
- Filler and repetition detection, which forced alignment to a clean transcript can't see on its own.
Log in or sign up for Devpost to join the conversation.