speechlens

Find exactly where a speech goes wrong, and show the math.

Comment

Inspiration

In competitive speaking (interpretive reading, declamation, extemporaneous, persuasive oratory), judges balance vocal delivery against content, and their scores drift. Contestants lose points for delivery and get feedback like "work on your pacing", with no time, no evidence and no number. We wanted a system that says "between 12.3 s and 14.1 s you spoke 1.7x too fast, and here is the calculation."

What it does

SpeechLens compares a participant's recording with a good reading of the same text, finds the exact stretches where delivery departs from it, and explains each one.

  • Temporal grounding: word-boundary start/end timestamps for each flaw region.
  • Causal explanation: each region carries the evidence, the formula with the real numbers, and a fix.
  • Dashboard: upload audio and text, see baseline vs. participant pitch/energy/pace/pause overlaid, flaw regions shaded, and click a note to hear the moment.
  • Rubric: five scores (pacing, pauses, pitch, energy, clarity) with speech-type weightings.

It detects ten flaws: rushing, dragging, misplaced / missing / overlong pauses, monotone, rising endings (uptalk), trailing off, volume collapse and mumbling.

How we built it

1. A contrastive dataset engine. No dataset maps good vs. bad delivery of identical text, so we generate one. Given a baseline and its word alignment, we inject flaws at severity $s \in (0,1]$ using Praat PSOLA (pitch and duration) and signal edits. For example, rushing scales word durations by $1/(1+0.9s)$ and monotone compresses pitch toward the speaker median by $(1-s)$ in semitones. We render the audio from word/gap segments and read each label off the final audio, so ground-truth timestamps are measured, not predicted. Untouched audio is bit-exact. Builds are bit-reproducible.

2. Alignment. DTW on normalised MFCC + energy projects the reference's word boundaries onto the participant, then boundaries are refined against the energy envelope (pause length is the most visible cue). A Montreal Forced Aligner TextGrid importer is included for gold-standard baselines.

3. Speaker-agnostic features. Pitch in semitones relative to the speaker's own median, loudness relative to the recording's own level, and a global tempo factor. Because both recordings contain the same words, we compare word by word.

4. Detection and explanation. For each word we compute a deviation scaled by natural between-speaker variation:

$$ z = \frac{\delta}{\sigma}, \qquad \text{e.g. pace: } \delta = \log_2!\frac{d_{\text{you}}/d_{\text{ref}}}{s} $$

Runs of words with $|z| \ge 2.5$ become regions. Pauses are judged against the punctuation the text provides. Explanations are deterministic templates (no LLM), and a test enforces that the displayed $z$ always equals $\delta/\sigma$.

5. Scoring. Each category is $100\,e^{-12D}$, where $D$ is the mean excess deviation beyond one natural-variation unit. Weights are transparent heuristics.

Stack: Python, Praat/Parselmouth, librosa, SciPy, Pydantic, FastAPI, a single-file JS dashboard with Plotly (vendored, works offline), 118 pytest tests, Docker, GitHub Actions.

Challenges we ran into

  • Nondeterministic audio. Praat's time-stretching used unseeded randomness, so the "same" dataset differed run to run. Seeding Praat's RNG before every resynthesis made builds bit-identical.
  • Quiet words looked rushed. A global silence threshold clipped quiet words and produced phantom "rushing" flags. A local, adaptive threshold fixed it.
  • Inconsistent math. Our first explanations showed a z-score that did not match their own delta. We restructured them and added an invariant test.
  • Speaker-agnostic is hard. Terminal pitch on words ending in unvoiced consonants is meaningless, so we only make that claim on words with enough voiced signal.
  • Fake silence. Inserted pauses use synthesised room tone matched to each recording's noise floor, otherwise a detector could just find digital silence.

What we learned

Most of the value came from debugging the evaluation, not the model: reading false positives critically exposed several real bugs. We also learned how easily a benchmark you built yourself can flatter you.

Honest results and limitations

On a development benchmark (reference voice vs. two other simulated speakers, 32 recordings), flaw regions reached precision 1.0 / 0.88 and recall 0.82 / 0.75, with matched-region IoU 0.91 / 0.84 and median boundary error 12-18 ms. Flawless takes by other voices produced zero false regions, scores fell monotonically down the quality ladder, and word-alignment median error was 7-10 ms.

These are synthetic speakers with flaws injected by our own engine, and several constants were tuned on the same recordings, so they show the pipeline works, not that it is accurate on human speech. Low-severity flaws are intentionally not flagged, and the rubric weights are not calibrated to human judges. The repo includes speechlens evaluate-human and a sourcing guide to measure real-speech performance.

What's next

Evaluate on human recordings with annotated flaws, calibrate tolerances on several good readers, use MFA-aligned baselines, and calibrate the rubric against real adjudicators.

Built With

Share this project:

Updates

Submission history