Inspiration

Anyone who has competed in speech and debate knows the ballot. You deliver a piece you rehearsed for weeks, and the judge writes "work on pacing" or "more vocal variety". It is not wrong, but you can't act on it. Where was the pacing off? In the opening line, or in the climax? How much too fast? Compared with what?

Judges in Interpretive Reading, Declamation, Extemporaneous and Oratory have an extremely hard job. They listen to tone, cadence, volume, pauses and content at the same time, in real time, and turn all of it into a score. Two good judges often disagree, and the participant loses points without knowing exactly why.

When I looked at the problem as an engineer, the gap was clear. There are excellent datasets for speech recognition (what was said), but almost nothing for speech delivery (how it was said). No public dataset pairs a good rendition of a speech with a spectrum of bad renditions of the exact same words, with the flaws marked in time. Without that, you cannot build or measure a system that says "your pause after country was 0.9 seconds too short".

That became the inspiration for ProsoDiff: treat speech as a time series, compare it with how the same text can be delivered well, and turn every deviation into something specific, located and explainable. The aim was a fairer judge's assistant and a better coach.

What it does

ProsoDiff takes a participant's recording and a transcript and answers four questions for every problem it finds: where it happened, what went wrong, by how much, and why it matters.

1. It compares like with like. The participant is force-aligned word by word to good renditions of the same text. These can be an ensemble of five reference voices, a real archival recording, the user's own ideal reading, or an ideal reading synthesised from any transcript. Every word is then compared with the same word in the reference.

2. It measures in units of natural variation. Two excellent speakers never deliver a sentence identically, so "0.4 seconds longer" means nothing on its own. ProsoDiff learns, from good speech only, how much each feature naturally varies between good speakers, and expresses each deviation as a z-score: z = (Δ − μ) / σ. A z of 6 means "six times further from the reference than good speakers ever drift".

3. It grounds flaws in time. Where the deviation passes a calibrated threshold, ProsoDiff cuts out a flaw region with exact start and end times. It covers twelve flaw types in six categories:

Category	Flaws detected
Pacing	rushed, dragging
Pausing	rhetorical pause skipped, hesitation pause inserted
Intonation	monotone (local or whole speech), erratic pitch, upspeak
Volume	trailing off, shouting
Clarity	mumbling
Fluency	"um/uh" fillers, restarted words

4. It explains each flaw causally. Every region gets a structured explanation: the measured values in natural units, the exact formula with its natural σ and z, the acoustic mechanism and why listeners perceive it as a flaw, and a measurable coaching target. For example:

Muffled articulation on "the solicitation of Japan was" (37.01–39.31 s). High-frequency tilt −39.5 dB vs +1.3 dB for the baseline. The 2–8 kHz energy that carries consonant bursts is missing, so word-distinguishing consonants (t, k, s, f) are lost. Fix: open the jaw and finish word-final consonants.

5. It scores the delivery on a six-criterion rubric (pacing, pausing, intonation, volume, clarity, fluency) and combines them into a 0–100 score with a letter grade.

6. It shows everything in an interactive dashboard. You upload audio and a transcript, or pick from the dataset, and get:

a five-panel time-series overlay (flaw lanes, pitch, energy, speaking rate, spectral centroid), where the baseline is re-timed onto the participant's clock so you compare delivery rather than speed;
flaw cards with A/B audio, so you hear the flawed words next to the same words delivered well;
a transcript heat-map, a word-by-word table of every z-score, and JSON, CSV and HTML exports.

It also ships with a contrastive dataset: 199 audio files, 214 minutes and 335 exactly labelled flaws. It includes the archival 1961 recording of John F. Kennedy's "ask not" line, mirrored with flaws injected into his own voice.

How I built it

ProsoDiff is about 4,300 lines of Python in eight stages: dataset construction, forced alignment, feature extraction, contrast, calibration, temporal grounding, explanation and scoring, with a Streamlit dashboard on top.

The dataset: making the answer known

The core problem was ground truth. To measure whether a system finds a flaw at the right moment, you need to know exactly where that flaw is. Archival recordings come with no word timing, and hand-labelling hundreds of flaws to the millisecond would be slow and inconsistent.

My solution was to make the answer known by construction. I rendered three public-domain speeches as deliberate ideal deliveries with the open Kokoro-82M text-to-speech model, using seven voices across US and UK accents and both sexes:

Lincoln's Gettysburg Address
two passages of Kennedy's Inaugural
the opening of Roosevelt's "Day of Infamy"

The key trick was to patch the model's ONNX graph to expose its internal duration predictor. Its per-token durations are integer frames of 600 samples, and they sum exactly to the waveform length. That gave me sample-exact boundaries for every phone and every word.

On top of those baselines I wrote twelve flaw injectors. Tempo and pitch changes use Praat's PSOLA, which preserves voice quality. The other flaws use gain envelopes, a Butterworth low-pass for mumbling, silence edits, synthesised fillers and syllable restarts. Each injector replaces an interval and applies an affine time map to every word, every phone and every previously written label:

t < a: t      a ≤ t ≤ b: a + (t − a) · L′/(b − a)      t > b: t + L′ − (b − a)

So flaws can be stacked freely while every timestamp stays exact. One intensity knob drives each injector, and from it I built a five-tier gradient from "near perfect" (one mild flaw) to "botched" (seven severe flaws plus a whole-speech flaw). I added 72 single-flaw probes to measure each detector alone, and controls (pink noise at 20 and 10 dB, −18 dB gain, two voices that never influence any parameter) that the system must not flag.

For the real-speaker requirement, I added the archival JFK recording. It has no ground-truth timing, so I force-aligned it to five synthetic readings of the same sentence and mirrored it with the same injectors in his own voice. Inserted fillers are pitch-shifted to his median F0.

Every file ships with four label formats: JSON, word and phone timing, a Praat TextGrid and an Audacity label track. The whole dataset regenerates bit-identically from seeded randomness.

Features: speaker-agnostic by design

Everything runs on a 10 ms grid:

Pitch comes from Praat's autocorrelation tracker, cleaned of spikes and octave errors, and expressed in semitones relative to the speaker's own median. A bass and a soprano reading identically produce identical contours.
Energy is measured in dB relative to the speaker's own speech level.
Spectral features come from a 512-point FFT: the spectral centroid and the high-frequency tilt 10·log10(P(2–8 kHz)/P(0.1–2 kHz)), the main clarity cue.
MFCCs (13 coefficients) drive alignment and disfluency matching.
Timing: articulation rate from the phonemised syllable count, and pause intervals from an adaptive energy gate.
A Wiener-style denoising front end is a no-op on clean audio and recovers the silence gate and spectral tilt on noisy recordings.

Alignment: TTS-DTW with a garbage model

Forced alignment uses dynamic time warping between the participant and a reference whose timing is known.

Speech-only frames. Silent frames are removed first, so inserted or deleted pauses cannot bend the warping path.
Features. MFCCs are combined with a speaker-independent energy and voicing envelope.
Garbage model. I extended DTW so the participant can produce extra material (an "um", a false start) at a flat cost instead of forcing it onto the next word. The recursion is compiled with numba.
Refinement rules snap word edges out of silence, split words that contain an unexpected internal pause, and snap onsets after long pauses to the audible start.

On unseen voices, 94 % of word boundaries land within 50 ms, with a median error of 10 ms.

Calibration: few false alarms first

This was the most important design decision. Each of the five ensemble voices in turn plays participant against the other four. That gives the natural spread σ of every feature, from robust statistics (MAD and the 90th percentile). Then, Neyman–Pearson style, every detector rule gets the most sensitive threshold that still produces at most 0.1 false regions per minute on good speech. A feedback tool that criticises good delivery quickly loses its users' trust.

Grounding, explanation and scoring

Span flaws use hysteresis region growing on words: seed at z ≥ T, grow at ≥ 0.6 T, then trim the edges with unsmoothed values so regions start and end on exact words.
Pause flaws are tested on each inter-word gap.
Fillers and restarts use structural detectors. A filled pause is a silence-flanked voiced island with almost no spectral change and level pitch. A restart is a fragment whose sounds recur at the next onset.
Explanations come from templates filled with the measured deltas.
The rubric combines continuous exceedance (consistency across all words) with region penalties weighted by strength and length, through 10·exp(−(1.6E + 0.9R)) per criterion.

Stress testing

The evaluation script runs every flawed and control file through the full pipeline against a leave-one-speaker-out ensemble, so the system never sees the participant's own voice in its baseline. A detection counts as correct only when its category matches a label and it overlaps the true interval with an IoU of at least 0.3. Boundary flaws get a ±0.25 s tolerance. Whole-speech flaws count as found when same-type regions cover at least 40 % of the speech. The script reports precision, recall and F1 per set, per flaw type, per intensity, per speech and per voice; onset and offset errors; score against severity tier; false alarms per minute on good audio; and alignment error against the exact ground truth. It writes everything to CSV, JSON and figures.

Dashboard and reproducibility

The dashboard is Streamlit with Plotly. Dependencies are pinned, models are verified by SHA-256, all randomness is seeded, and the evaluation re-runs a file to assert identical output. The technical report and README fill their numbers in from the evaluation results, so the documents can never drift from the measurements.

Challenges I ran into

There was no ground truth to start from. The biggest early obstacle was how to measure temporal precision at all. Exposing the TTS duration predictor solved it, but only after I traced the ONNX graph node by node to find the rounded duration tensor, and checked that the durations summed to the audio length.

My first evaluation was humbling. The first full stress test produced 11.8 false flaw regions per minute on good speech, and precision was 0.29. Comparing against a single reference voice meant every quirk of that voice became a "flaw". I fixed this in three steps:

An ensemble of five references, combined with a median, so no single voice can create a flaw.
Within-speaker standardisation, so each voice's dynamic range is factored out.
Calibrating every threshold to a false-alarm budget.

False alarms fell to 1.2 per minute and gradient precision rose to 0.70.

A subtle pitch bug hid half my data. Pitch-based detectors were mysteriously weak. Praat's frame centres sit at k·dt + dt/2, and my rounding mapped them onto the 10 ms grid with collisions, so every other frame looked unvoiced. A one-line resampling fix made the erratic-pitch and upspeak detectors usable for the first time.

DTW kept swallowing fillers. An injected "um" before a word was often absorbed into that word, which then looked long and slow instead of disfluent. I added the garbage model, then a split-and-match step that cuts a word at unexpected internal silences and keeps whichever piece matches the reference better. In the end I added structural filler detection that doesn't depend on cross-speaker matching at all.

My synthetic fillers weren't realistic. Raw TTS "um" tokens have a speech-like pitch contour, so no honest filled-pause detector should accept them. Real filled pauses are sustained and level-pitched, so I flattened and lengthened the injected fillers to match how humans actually produce them.

A tiny alignment bias broke upspeak. Upspeak recall sat at 0.07 for a long time. The cause turned out to be word boundaries: the aligner trimmed the rising tail of the final word, which is exactly where upspeak lives. Fixing onset and offset snapping around pauses raised upspeak recall to 0.79.

Real speech is not synthetic speech. When I added the JFK recording, onsets after long pauses came out 150–200 ms late. Fixing that naively hurt synthetic accuracy, and I found out why: the TTS ground truth itself sits about 80 ms after the audible onset. I settled on snapping only gross misses (over 120 ms), which fixed the real audio without distorting the benchmark.

Noise was brutal. Before denoising, good speech at 10 dB SNR scored 43 out of 100, because noise filled the pauses and flattened the spectral tilt. The Wiener-style gate raised that to about 82, with 20 dB noise becoming practically harmless.

My environment was locked down. I built everything in a sandbox where most archival audio sources were unreachable. The real JFK clip came from the public-domain sample shipped with open-source speech recognition projects on GitHub. It is only 11 seconds, which limits how much real-speaker evidence I could gather.

Accomplishments that I'm proud of

A dataset whose labels are exact by construction. The exposed duration predictor plus time-map label propagation means every one of the 335 flaws has a known start and end, and the whole dataset regenerates bit-identically. I think this is the most reusable idea in the project: any new text gets a labelled "bad-mirror" gradient for free.
Honest, calibrated detection. Thresholds are chosen against a false-alarm budget on good speech, not tuned to look good on the flawed set. Held-out voices score 93.6 on average, and good audio overall averages 90.8.
Results that hold up under measurement:

Measure	Result
Gradient precision / recall / F1	0.70 / 0.73 / 0.72
Flaw type correct when found	100 %
Median flaw onset error	49 ms
Median span IoU	0.88
Whole-speech flaws found	12 / 12
Score vs severity (Spearman ρ)	0.86
Mean score, near-perfect → botched	87.5 → 84.7 → 79.6 → 65.2 → 44.1
Word boundaries within 50 ms (unseen voices)	94 %
Identical output on re-run	yes

Explanations a judge could put on a ballot. Each flaw states its numbers, its formula and its z-score, explains the perceptual mechanism, and ends with a target someone can practise toward, such as "hold about 0.7 s of silence after world".
A real voice in the loop. Kennedy's own voice, mirrored with flaws, shows that recall carries over to a human speaker (0.68). And the comparison of his real delivery with the synthetic ideal shows exactly why reference choice matters.
Reproducibility end to end. Pinned dependencies, checksum-verified models, seeded randomness, an automatic determinism check, and documents that pull their numbers from the latest evaluation.

What I learned

Data engineering is the real work. I spent more effort designing how to know the answer than designing the detector, and that investment paid off every time. Each improvement could be measured instead of eyeballed.
Calibrate for trust, not for recall. It is easy to build a detector that finds every flaw and flags half of every good speech too. Fixing the false-alarm rate first and maximising sensitivity within it produced a tool whose feedback I would actually believe.
Normalisation is what makes "speaker-agnostic" real. Semitones relative to the speaker's own median, dB relative to their own level, within-speaker standardisation and a multi-voice ensemble are what let a bass, a soprano, a British and an American voice land on the same severity gradient.
Small bugs hide in the joins between tools. The two biggest jumps in quality came from a frame-resampling off-by-half and a word-boundary bias, not from new algorithms. Plotting the raw contours under each detector was the best debugging tool I had.
Classic signal processing still earns its place. PSOLA, FFT spectral tilt, DTW and robust statistics, combined carefully, gave interpretable results whose rationale can be written down as a formula. For a feedback tool, that interpretability matters as much as accuracy.
Report weaknesses plainly. Real-voice precision is 0.39, mild erratic pitch and trailing-off are often missed, and very noisy long recordings degrade. Writing those down made the project's next steps obvious.

What's next for ProsoDiff

Calibrate on real humans. The single most valuable step is to record 30–50 human readings (debate students are ideal), have several judges score them, and recalibrate σ and the thresholds on human voices. The calibration code already accepts any reference set. The target is real-voice precision of at least 0.7 and a correlation of at least 0.7 with judges' scores.
Stronger cross-speaker alignment. Replace MFCC features in the alignment with speaker-independent phone posteriors (for example from a wav2vec2-CTC model), so human-to-human alignment is as robust as synthetic-to-synthetic.
Learned detectors on interpretable features. The z-score series are compact, meaningful features. A small sequence model trained on the labelled gradient could raise recall on mild flaws, while the calibrated rules remain the explanation layer.
Emphasis and content. Detect which word carries the stress and compare stress placement with the reference, a key skill in interpretation events. Also flag skipped, added or substituted words alongside delivery.
A coaching product. A hosted app with browser recording, a library of model readings, progress tracking and printable timestamped ballots, then a judge-assist mode for live rounds and batch scoring for leagues and schools.
Beyond competitions. The same contrastive machinery fits presentation training, read-aloud fluency assessment, prosody practice for language learners, and automatic prosody quality checks for text-to-speech systems.
A larger real corpus. Public-domain presidential and parliamentary archives plus consented debate recordings, each mirrored with the existing injectors, would turn ProsoDiff's dataset into a public benchmark for delivery analysis. That is the dataset I wished existed when I started.

Built With

analytical-path
openai/whisper
speechapi
whisper.cpp