Speech Delivery Coach

Inspiration

I first built AI Dataset Maker, a pipeline that turns long Uzbek audiobooks into clean, verified TTS training data. Its core idea is to force-align every word and then let a second model check each clip before it's kept. Track C showed me the same problem from the other side. Speech judges disagree, and speakers get feedback like "work on your pacing", which says neither where nor why. If two readings of the same text can be aligned word by word, we can measure exactly where a delivery drifts, and say so in numbers.

What it does

You pick a reference delivery (Roosevelt, Kennedy, Reagan, or your own) and record or upload a reading of the same text. One analysis drives every view:

Style profile: your overall tempo, pause length and pitch range vs the reference
Timestamped flaws: rushed, dragging, long pause, monotone, mumble, stutter, unclear articulation, each with its numeric cause, e.g. "3.96–4.94 s · speech rate 9.2 vs 6.2 syl/s (+47%, z = +11.2)"
A coaching tip per flaw with a concrete target, and a practice plan ranked by the points each issue costs
Word heatmap, A/B listening of the same words (reference, then you), and a PDF report
A reproducible 0–10 rubric: Pace, Pausing, Expressiveness, Volume & clarity, Fluency

How we built it1. Contrastive dataset (63 files). Three public-domain speeches (FDR 1941, JFK 1962, Reagan 1986) are the "good" baselines. Each is mirrored by 6 flaw types × 3 severities, plus a near-perfect and a botched composite. Flaws are injected by signal processing: PSOLA time-scaling with pitch preserved, Praat pitch-tier flattening, room-tone pauses, band-pass + gain, and repeated onsets. Every operation carries a time map, so ground-truth timestamps are exact to the sample.

2. One word-aligned delta table. Both readings are force-aligned to the same transcript with Meta's MMS model, with a wildcard token between words to catch sound that isn't in the text. Word $k$ in one reading is word $k$ in the other. Per word: pitch in semitones relative to the speaker's own median,

$$\text{st}(t) = 12 \log_2 \frac{F_0(t)}{\operatorname{median}(F_0)},$$

energy in dB relative to the speaker's median, articulation rate, pauses, spectral centroid, and MFCC shape (DTW).

3. Style first, then flaws. The medians of the deltas form the participant's style profile, and local flaws are measured against that style, so a naturally slower voice isn't flagged everywhere. Each deviation is z-scored against the pair's own measurement noise:

$$z_k = \frac{d_k - \operatorname{median}(d)}{1.4826 \cdot \operatorname{MAD}(d)}.$$

A word is flagged only if an effect-size threshold and $|z| > 1.5$ both pass. Flagged words merge into regions, and a clean-up pass enforces one cause, one region.

4. Rubric and coaching. Each region adds a penalty $m \cdot \left(0.5 + 0.5 \min(1, \tfrac{\Delta t}{5\,\text{s}})\right)$, and profile habits add their own. Each criterion scores $10 \cdot \max(0, 1 - \sum \text{penalties})$. There's no sampling and no LLM, so the same input gives the same score every time. The practice plan is simply the rubric items that cost the most points.

Stack: Python, PyTorch/torchaudio (MMS_FA), Praat via parselmouth, librosa, Gradio, Plotly, fpdf2.

Challenges we ran into

Whisper hallucinated on the 1940s–80s recordings (it repeated "Bye." over applause), so I used the official transcripts and let the aligner handle timing.
False alarms from my own fixes. Breaths looked like stutters, muffled words looked like silence, and a global MFCC normalization made every word look mispronounced after a single inserted pause. Each fix came from reading the stress-test output: counting only excess over the baseline, a noise-floor check for pauses, and per-word cepstral normalization. Precision rose from 56% to 66% with recall unchanged.
Fairness across voices. The first version flagged a slower speaker everywhere. The style profile fixed that.

Accomplishments that we're proud of

A stress test where the detector never sees the labels: it finds 81% of injected flaws (89% at severity 2) with 0 false alarms on the clean speeches.
Pauses localized within ~0.03 s, pace within ~0.05 s.
Scores fall with severity, 10.0 → 9.0 → 5.3 (clean → near-perfect → botched), and are fully reproducible.
Every number on screen traces back to one delta table: profile, flaws, tips, plan, heatmap, A/B clips and PDF all agree.

What we learned

Exact labels matter more than dataset size. Because every flaw was injected deliberately, I could measure the detector honestly instead of guessing. I also learned to report what doesn't work yet: single stutters (4/12) and articulation (4/9 on a reversed-word probe) are the weakest classes, and thresholds were tuned on this same dataset, so the numbers are optimistic.

What's next for Speech Delivery Coach

A held-out speaker and self-recorded flawed readings, to test beyond synthetic flaws
More diverse voices (gender, accent, recording quality)
The same "verify before you trust" pipeline in other languages, starting with Uzbek, where AI Dataset Maker began

AI disclosure: Claude Code was used as a coding assistant. Details are in the README.