speech analytics

Judging competitive speech (interpretive reading, declamation, oratory) is subjective. Two judges can score the same delivery very differently, and participants get vague feedback like "work on your pacing". There is also no dataset that pairs a strong delivery with a spectrum of flawed deliveries of the exact same text. We wanted a system that says where a delivery goes wrong, to the second, and why, in numbers.

What it does

Upload a recording and its transcript, pick an ideal reference reading of the same text, and the dashboard:

aligns both recordings to the transcript at word level,
overlays pitch, loudness and speech-rate contours of the reference and the participant,
highlights flaw regions with exact start and end times and a severity from 1 to 5,
explains each flaw with the measured numbers (for example: speech rate +38% vs the reference and pauses -60%, so the phrase sounds rushed),
gives an overall score and sub-scores for pacing, pauses, pitch, loudness and clarity from a documented rubric.

Comparison is speaker-agnostic: pitch is expressed in semitones relative to each speaker's own median, and loudness in dB relative to each speaker's own mean, so a different voice is not mistaken for a flaw.

How we built it

Dataset. We sourced [N] real recordings of public speeches as "ideal" baselines (licences listed in SOURCES.md) and transcribed them. For each baseline we generated mirrored flawed versions of the same text with injected flaws: long pause, missing pause, rushed, dragged, monotone, mumble and stutter, each at five severity levels from L1 (almost imperceptible) to L5 (obvious), plus mixed-flaw clips and control clips (pitch shift, gain change, resampling) that must trigger no flaws. Every label is computed from the baseline's word alignment, so start and end times are exact. [We also recorded N human flawed readings as a separate test set.]

Alignment and features. Word timestamps come from forced alignment with torchaudio's MMS_FA model. Per word we extract duration, pause before the word, F0 (Praat via Parselmouth), energy, local speech rate and harmonics-to-noise ratio.

Detection. Both recordings are matched word by word. For each word we compute deviations in rate, pause, pitch and energy, remove the clip's global offset, and convert each deviation to a robust z-score:

$$z_i = \frac{d_i - \tilde d}{1.4826 \cdot \mathrm{MAD}(d)}$$

Words with large z-scores are merged into flaw regions, typed by rule (for example a long word with unchanged pause is "dragged", a gap before a word is a "pause"), and given a severity. Explanations are templates filled with measured values. No language model writes them, so every number is traceable.

Dashboard. Streamlit and Plotly, with time-aligned contours, shaded flaw bands, explanation cards and audio players. Everything uses fixed seeds and pinned dependencies [and runs from a Dockerfile].

Challenges we ran into

Ground truth timing. Editing audio shifts later timestamps, so labels had to be recomputed in the new clip's timeline after every edit.
Pause or drag? A long gap and a stretched word look similar in duration, so we separate them using pause-before-word versus word duration.
Speaker differences. A deeper or quieter voice must not count as a flaw, which required per-speaker normalization.
Alignment edge cases. Hyphens and digits in transcripts broke the aligner (a blank token reached the target sequence), so we added one shared tokenizer and clear errors for unsupported text.
Honest baselines. Early on we used synthetic voices as stand-ins, then replaced them with real recordings, because flat synthetic pitch makes pitch flaws meaningless.

Accomplishments that we're proud of

A contrastive dataset with exact, programmatically verified time labels and a five-level gradient down to near-perfect delivery.
Control clips that should trigger nothing: [N] false alarms per minute on controls.
Region-level [F1 = X] on the test split, mean timestamp error of [X] s [and F1 = X on human-recorded flaws].
Plain-language explanations backed by measured numbers.

What we learned

Dataset quality matters more than model complexity. Well-labelled contrasts and per-speaker normalization did more for accuracy than any feature we added. Near-perfect deliveries (L1) are by far the hardest to detect, and we report results per level instead of hiding them in one average. Reproducibility also needs discipline: fixed seeds, pinned versions and no hidden state.

What's next for speech analytics

More human-recorded flawed readings across several speakers and accents.
Content-aware feedback that ties acoustic stress points to meaning (emphasis on the right words).
Learned severity models trained on judges' scores, with the rule-based explanations kept for transparency.
Live feedback while practicing, and support for more languages.