Rostrum: Project Story

Inspiration

Judging competitive speech is subjective. Two judges at the same Declamation round can weight pace, pausing and emphasis differently, and "good delivery" has no single acoustic definition. The obvious machine-learning answer is to train a regressor on human scores. That bakes the subjectivity in, and it can't tell a speaker where in the recording the problem is, only that a number came out low.

We wanted a system that could say something like "between 41.21 s and 45.88 s you rushed, and here are the exact numbers that show it." Three observations shaped the design:

There is no single correct reading. A declaimer isn't supposed to mimic the original orator, so any legitimate baseline has to be a distribution of skilled readings. A single reference would punish valid interpretive choices.
Most of what looks like a delivery difference isn't one. A rate of 6 syll/s means nothing in the abstract. Word choice, voice and recording channel all change the acoustics without changing the delivery.
Prior work stops short of this. Systems like VoiceCoach name the lack of negative delivery examples as future work. A negative example with exact timestamps is the thing nobody has, and it can be manufactured by intervention.

So we set a constraint for ourselves: no language model in the judging loop. Every number and every timestamp in every explanation is produced by signal processing, and language is only the template that wraps them.

What it does

Rostrum evaluates how a speech is delivered, not what it says. Given a recording and its transcript, it:

force-aligns every word with the Montreal Forced Aligner (word tier and phone tier);
extracts eleven feature families on one 10 ms / 25 ms grid;
compares each word against an envelope of skilled readings of the same text;
grounds each deviation to a start and end timestamp;
explains it with the numbers that produced it, and states a counterfactual fix.

A real card looks like this:

[41.21 - 45.88 s]  Pace / rushing  (L3, confidence 0.93)

  Across "we can not dedicate, we can not consecrate" the speaking rate reaches
  6.4 syll/s, 4.1 sigma above the skilled readers' 3.9 syll/s here;
  the boundary pause before it fell from 0.82 s to 0.06 s (-3.6 sigma).

  Context: repeated phrase ("we can not", 3 times in this text), mid-clause.
           6 of 6 skilled readers paused at least 0.25 s here.

  Fix:     slow this stretch to about 3.9 syll/s and hold the pause.

Eight flaw labels, six rubric categories

Category	Labels	Primary evidence
Pace	`rushing`, `dragging`	local articulation rate
Pausing	`missing_pause`, `hesitation_pause`	gap duration against a text-derived expected-pause label
Intonation	`monotone`	phrase F0 range and standard deviation, in semitones
Emphasis	`misplaced_stress`	rank correlation of within-phrase prominence
Energy	`low_energy`, `fading`	relative energy and its slope
Clarity	`mumbling`	spectral tilt, HNR, CPP, MFCC, same channel only

Three operating modes

A delivery evaluator that gives confident numbers for audio that doesn't match its transcript is worse than useless. Rostrum resolves a mode before any comparison, and shows it in every output:

Reference: the transcript matches a library text (similarity $\ge 0.95$) with at least 4 readers. The text's own word-by-word envelope is used.
Norm: no library text matched. Pooled norms are used, channel-sensitive features cannot fire, and clarity is dropped from the mean.
Reject: ASR word error rate against the transcript is above $0.25$. No score is given, and the banner quotes what the recogniser actually heard.

The dashboard

make app serves a Streamlit frontend on :8501 and a FastAPI service on :8000. It accepts an audio file and a transcript and renders:

a time-series overlay of F0, energy and rate against the envelope band, with flaw regions shaded;
a transcript tinted word by word by deviation;
a waveform with A/B playback (the participant's span against the same words read by a skilled reader);
a six-category rubric radar with a confidence band;
a one-click PDF report.

How we built it

1. Removing the confounds

Three things are fixed so that a measured acoustic difference can only come from delivery.

Confound	Removal
Wording	Compare word $i$ to word $i$ across readings of the same text.
Voice	Compare only in speaker-normalised units.
Channel	Spectral, cepstral and voice-quality features are compared only within one recording channel. Crossing channels raises an exception rather than a warning.

The speaker-normalising transforms are:

$$ f_{\text{st}} = 12\,\log_2!\left(\frac{f_0}{\tilde f_{0,\text{speaker}}}\right) $$

Energy is expressed in dB relative to the speaker's voiced median and then z-scored. Rate is reported both in absolute syll/s and as a ratio to the speaker's own articulation rate. MFCC and spectral features use per-utterance CMVN. Pitch variation uses the standard deviation in semitones, because the Hz-ratio pitch-variation quotient is documented to miss monotone delivery in female voices and would make the monotone label sex-biased.

2. A dataset built by intervention

Sourcing. TED is licensed CC BY-NC-ND, and NoDerivatives forbids exactly what the project must do, which is make edited flawed versions. The downloader refuses any ND licence and refuses TED outright. Instead we used LibriVox collections in which many volunteers read the same short text.

Tier	Source	Texts
Gold	FDR's own Day of Infamy recording (Wikimedia Commons, public domain)	`s3_fdr_infamy`
Silver	LibriVox: Gettysburg Address (16 readers), Dickinson (17), Lampman (14), Sonnet 18 (14)	`s1`, `s5`, `s6`, `s7`
Bronze	consented team takes, CC BY 4.0, structurally test-only	all

The result is 62 clean readings (94.7 min) and 266 injected variants, for 328 audio files, 683.3 minutes and 364 temporal events. Every file carries a licence row and a saved licence statement, and make provenance fails the build if either is missing.

Injection. A single scalar severity $s$ drives every flaw formula, so the gradient is one axis rather than seven unrelated knobs. The five rungs are $s \in {0, 0.15, 0.35, 0.60, 0.90}$ for L0 to L4. For example:

$$ \text{rushing: } \ \text{factor} = 1 - 0.45\,s \qquad \text{dragging: } \ \text{factor} = 1 + 0.8\,s $$

$$ \text{missing_pause: } \ \text{keep} = 1 - 0.95\,s \qquad \text{hesitation: } \ d = 0.3 + 1.7\,s \ \text{s} $$

$$ \text{monotone: } \ f_{\text{new}} = m\left(\frac{f}{m}\right)^{e},\ \ e = 1 - 0.9\,s \qquad \text{low_energy: } \ g_{\text{end}} = -18\,s \ \text{dB} $$

Edits use TD-PSOLA through Praat. A listening study places it level with a neural editor, and its Praat implementation is sample-exact, which matters for what follows. Inserted silence is looped room tone harvested from the recording's own pauses, never digital zeros, because a detector could trivially cheat on exact zeros and learn to find editing rather than hesitation.

Exact labels. Each operation returns a piecewise-linear old-to-new TimeMap. A flaw's ground-truth span in the edited file is the clean span pushed through that map:

$$ [t_a',\, t_b'] = \big[\,\mathcal{T}(t_a),\ \mathcal{T}(t_b)\,\big] $$

Splicing consumes no length at the joins, so the new region length equals the replacement length exactly. The test suite asserts that every later word boundary shifts by exactly the computed delta to within 1 ms. In Pearl's terms this is a do-operation, so on the synthetic portion the acoustic delta at a known span is causally attributable to the injected flaw.

Two hard QC gates reject files rather than keep them:

Gate	Test	Rejects when
Re-alignment	Re-run MFA on the edit and compare to the clean alignment pushed through the time map	any word boundary moved $> 50$ ms
UTMOS	Naturalness MOS against the clean source	drop $> 0.30$ at L1, $0.60$ at L2, $1.00$ at L3, $1.50$ at L4

The first protects the label, since a file whose boundaries moved has timestamps nobody can trust. The second protects the construct, since at L1 the edit should be inaudible and a large naturalness drop means the detector would learn to find resynthesis instead of rushing.

mumbling is deliberately never synthesised. A low-pass filter models a bad microphone, not bad articulation, and would also be a channel manipulation. It is human-recorded only.

3. Feature extraction

Everything lives on one frame grid, $10$ ms hop and $25$ ms window, with center=False so frame $i$ starts at exactly $i\cdot\text{hop}$.

Pitch: Praat autocorrelation via parselmouth, with a two-pass speaker-adaptive range (pass 1 at 50–700 Hz, pass 2 at $[0.75\,Q_1,\ 1.50\,Q_3]$). It is cross-checked against CREPE, and frames where the two disagree by more than 50 cents are excluded from the statistics.
Rate: syllable nuclei come from vowel phones on the MFA phone tier, not a dictionary lookup, with the de Jong–Wempe method as an independent cross-check.
Pauses: MFA gaps refined with Silero VAD, with a 250 ms threshold and a text-derived expected-pause label (boundary, minor, none).
Spectral, MFCC, voice quality, eGeMAPS: tilt, centroid, flux, rolloff, band ratios, CMVN'd MFCCs with deltas, HNR, jitter, shimmer, CPP, and the eGeMAPS v02 functionals.
Prominence: a continuous-wavelet stress detector. We build a composite signal

$$ c(t) = 1.0\, z_{f_0}(t) + 0.5\, z_{E}(t) + 1.0\, z_{\text{dur}}(t) $$

and take a Mexican-hat CWT. A scale $a$ responds to period $T = a\cdot\Delta/f_c$ with $\Delta = 10$ ms and $f_c = 0.25$, so 61 scales from $a = 2.5$ to $a = 80$ span $T = 0.1$ s (a short syllable) to $3.2$ s (a short phrase). Each scale is divided by its own standard deviation before summing, so the high-energy long scales can't drown the syllable scales. A word's prominence is the maximum of the summed response inside it.

4. The reference envelope

For each unit $w$ (word, gap or phrase), feature $k$ and $R$ skilled readings:

$$ \mu_{wk} = \operatorname*{median}{r}\, x{wk}^{(r)}, \qquad \sigma_{wk} = 1.4826\ \operatorname*{MAD}{r}\, x{wk}^{(r)}, \qquad \hat\sigma_{wk} = \max!\big(\sigma_{wk},\ \sigma^{\text{floor}}_{k}\big) $$

We use median and MAD because $R$ is only 4–20. One reader who sneezed mid-sentence would shift a mean and inflate the spread enough to hide a real flaw elsewhere. The constant $1.4826$ makes $1.4826\cdot\text{MAD}$ a consistent estimator of $\sigma$ for normal data, so z-scores keep their usual meaning.

The envelope has no learned parameters, so there is nothing in it that could memorise a speaker.

The sigma floor $\sigma^{\text{floor}}_k$ is the 25th percentile of sigma across all units of all training texts. It stops a unanimous unit (every reader pauses 530 ms after "perish from the earth") from turning a 40 ms difference into $z=-4$. It is also the leakage-sensitive part. fit_sigma_floors sees training texts only, and the held-out text receives floors fitted elsewhere.

5. Temporal grounding

Deviation and combined word deviation:

$$ z_{wk} = \operatorname{clip}!\left(\frac{x_{wk}-\mu_{wk}}{\hat\sigma_{wk}},\ -12,\ 12\right), \qquad D_w = \sqrt{\sum_k \lambda_k\, z_{wk}^2} $$

Because $D_w^2$ is a weighted sum of squares, each feature's share

$$ \text{share}k = \frac{\lambda_k z{wk}^2}{D_w^2} $$

is an exact additive attribution. The evidence table in each card is that decomposition, not a post-hoc approximation.

Each label becomes one signed trigger score, oriented so that positive always means further toward this flaw. For example, rushing fires on $+z[\text{rate}]$ and dragging on $-z[\text{rate}]$, so region formation is label-independent. misplaced_stress is the exception. It uses the Spearman correlation $\rho$ between the participant's within-phrase prominence ranking and the readers' consensus ranking, and fires below $\rho = 0.5$. Which word should be stressed is a property of the text and the readers, not of the acoustics, so no absolute threshold could work.

Region formation runs in four steps:

3-unit median smoothing, so one misaligned word can't cross the line.
Hysteresis: enter at $\tau_{\text{in}}=2.5$, stay open while above $\tau_{\text{out}}=1.5$. This is Canny's thresholding applied to a prosodic deviation curve, and it stops a score hovering near 2.5 from producing five regions where a listener hears one.
Merge same-label regions less than 200 ms apart and drop those shorter than 250 ms. The pause labels are exempt from the drop, because a pause flaw legitimately occupies one short gap.
Boundaries come from the alignment, never the score, so regions always name whole words.

Boundaries are then refined by ruptures PELT with an RBF cost over a $\pm300$ ms window. A refinement that moves a boundary more than 300 ms is discarded, and if PELT finds no change point the raw boundary is kept rather than nudged to the window edge.

6. Learned components, kept on a leash

Rostrum is deliberately not end-to-end. Learned parts exist only where they beat a rule, and none of them can override the explainable path.

Flaw-type classifier: HistGradientBoostingClassifier (200 iterations, random_state=0, early stopping off for bit-identical runs). The rule's label is always what is displayed. When the classifier disagrees with confidence $p > 0.8$, the card says so ("the rule reads this as dragging while the classifier reads it as low_energy at $p=0.87$") rather than silently substituting.
Severity calibrator: isotonic regression from deviation mass to $s$, monotone by construction, so more deviation can never mean less severity.
Thresholds and penalty: grid search on event-F1, fitted on training folds only.

7. Rubric scoring

Per category $c$, we integrate duration times mean excess deviation over its regions:

$$ M_c = \frac{1}{T_{\text{speech}}} \sum_{j \in c} \text{dur}j \cdot \overline{\big(|z| - \tau{\text{out}}\big)+}{\,j} \qquad S_c = 10\, e^{-k_c M_c} \qquad S = \frac{1}{6}\sum_{c} S_c $$

We use deviation mass rather than a region count because a detector that splits one flaw into two would otherwise make the reading look twice as bad, and mass is invariant to that split. The exponential maps $[0,\infty)\to(0,10]$ with no clipping, and a reading with no regions scores exactly $10$.

Calibration is closed-form. Since $S_c$ is strictly decreasing in $M_c$, the median score is the score of the median mass, and fixing the median single-flaw L2 file at $6.0$ gives:

$$ k_c = \frac{\ln(10/6)}{M_{\text{median}}} $$

A leave-one-reference-out confidence band recomputes the whole analysis with each reader dropped in turn. A wide band is itself a finding, since it says the references disagree about this text.

8. Causal explainability

We use three senses of "causal", each of which can be defended:

Interventional: the flaw was set on an otherwise identical recording.
Attributional: $D_w^2$ decomposes exactly across features.
Counterfactual: every card states the value at which the region would not have been flagged (the Wachter et al. form).

Every sentence is a template filled with computed numbers. A placeholder with no value raises TemplateError, so a half-rendered sentence cannot reach a user. The text-context module (repeated phrases, list items, clause position) only chooses which context clause a card prints, and never enters a z-score. The measurement doesn't depend on our opinion of the rhetoric.

9. Reproducibility

make reproduce regenerates every number from raw data. Seeds come from config and PYTHONHASHSEED=0. Outputs are rounded before hashing so cross-machine float noise can't break determinism, and caches are content-addressed by $\text{sha256}(\text{audio} + \text{config} + \text{alignment fingerprint} + \text{options})$. Every tunable lives in configs/, and parameter formulas are stored as strings so each label JSON records factor = 1 - 0.45 s and the resolved 0.595. The whole pipeline ships as one Docker image with MFA 3.2.1 baked in.

Challenges we ran into

Licensing was a build step, not a footnote. The brief suggests TED, but ND licensing makes edited derivatives illegal. We had to rethink sourcing around many-readers-of-one-text and write the refusal into the downloader itself.

mumbling can't be faked. Every cheap proxy (low-pass, noise) models the microphone, not articulation, and would violate our own channel rule. We left it out of the synthetic spectrum on purpose.

The channel confound is structural. Every clean reading comes from a different LibriVox speaker, so no recording has a same-channel reference. As a result, clarity cannot fire anywhere on this corpus. The trigger log records mumbling:no_same_channel_reference on every case, leaving 8 of the 9 triggers live. Measuring it properly needs clean and flawed takes from the same speaker and channel.

Alignment infrastructure. MFA pins Kaldi, OpenFst and pynini through conda-forge, and installing it beside librosa, torch and streamlit breaks one side or the other. We split the project into two environments. On a Windows dev machine there was no conda, no ffmpeg, and Smart App Control blocked unsigned native extensions. The Docker image was the only path verified end to end. Two readings (s7_lv09, s7_lv10) aligned 115 words against a 114-token transcript. We refused to silently shift every later index and left them out, since word $i$ must mean word $i$ across readers.

One text can't make an envelope. s3_fdr_infamy has a single clean reading, so it is permanently a Norm-mode text and is excluded from evaluation. The dataset LOTO runs over four folds, not five. A text that can't form an envelope also aborts the whole run with no checkpoint, so we had to name the viable texts explicitly.

Region geometry on the real corpus. This was the hardest and most humbling challenge. On the generated-table corpus, region formation looks healthy (pooled event-F1 $0.645$, recall $0.833$). On the 234 injected recordings of the real dataset it fails every pre-registered target:

Metric	Target	Achieved
Event-F1 @ 200 ms	$\ge 0.75$	$0.0083$
Onset/offset MAE	$\le 150$ ms	$506.7$ ms
Macro-F1 (type)	$\ge 0.7$	$0.4455$
Severity Spearman $\rho$	$\le -0.85$	$-0.0376$
False positives on clean (per min)	$\le 1$	$5.978$

That is 7 true positives against 1,656 false positives. Our diagnosis, checked rather than assumed, is that the features, the envelope and the labels are sound. Of 1,701 injected flaw words, 46.1% exceed $|z|>2.5$, so the deviation lands in the right place. The failure is geometry. Hysteresis flags only the subset of a flaw's words that cross the threshold, while the label covers the whole injected span (median 1.61 s). A correct region covering one word of a three-word flaw scores IoU $\approx 0.33$ and is counted as both a miss and a false alarm. The false-alarm rate has the same cause from the other side, because a 12–17 reader envelope drawn from different speakers has a wide enough MAD that ordinary interpretive variation crosses 2.5.

We chose to report this plainly, because the alternative was quoting the generated-corpus numbers as if they were the real result.

Accomplishments that we're proud of

A contrastive dataset where labels are arithmetic, not opinion. 328 label files, 364 events, 5 severity rungs, every timestamp computed from an exact TimeMap and verified by re-alignment and UTMOS.
Quality over quantity, enforced. Two QC gates reject files and log why, instead of quietly keeping them.
Clean forced alignment. 60 of 62 readings aligned, 0 out-of-vocabulary words, 0 files flagged by the 2% unaligned-word budget, and an unaligned fraction of 0.000 on every aligned file.
Validated pitch. Praat and CREPE agree within 50 cents on 93.93% of 1,383 commonly voiced frames, with a median difference of 12.78 cents and an octave-slip rate of 0.43%. An octave error would fabricate a 12-semitone deviation and a false monotone, so this is the evidence that the F0 the cards quote is real.
A severity-ordered rubric. The score is monotone decreasing in injected severity in every fold, including the barely-perceptible L1 rung, and clean files with no regions score exactly $10.000$.
Exact explanation. $D_w^2$ decomposes additively, every card has a counterfactual fix, and the same feature (gap_duration) produces opposite, correctly-directed advice depending on whether the text asks for a break.
No LLM in the judging loop, with a system that says which of its own numbers to distrust via mode badges, warnings and confidence bands.
Engineering discipline. 1,366 tests across 47 modules built to catch wrong numbers rather than crashes, determinism asserted by hashing, and one-command reproduction in a container.
Honest evaluation. We pre-registered targets before running, and reported the misses.

What we learned

Intervention gives you labels that observation can't. Setting a flaw on an otherwise identical recording turns a correlation study into a controlled experiment, and it makes ground-truth timestamps free.
Robust statistics matter at small $R$. Median and MAD are what let a handful of readers form a trustworthy envelope.
A learned component is safest when it can only disagree out loud. Letting the classifier annotate, but never overwrite, the rule's label keeps every displayed label traceable to a threshold crossing on a named feature.
Leakage hides in the boring parts. The sigma floor, the lambda weights and the $k_c$ constants are all fitted quantities. Making their fold-correctness structural (fitters that apply the filters themselves) is more reliable than trusting the caller.
Component tests are not system tests. The generated-table corpus tests region formation in isolation. Only the real corpus tells you whether the geometry survives contact with real speakers, and here it did not.
Detection precision and region overlap are different problems. A detector can be "right" about where deviation lives and still score near zero if its regions don't tile the labelled span.
A negative result is a result. Documenting why the grounding claim isn't yet supported end to end is more useful than a headline number we can't stand behind.

What's next for Rostrum

Fix the region geometry. This is the main item. We plan to (a) score and train against span coverage rather than per-word threshold crossings, (b) extend each detected core outward along the smoothed score until it decays (an IoU-aware extent policy, comparing smoothed and raw extension), and (c) tune $\tau_{\text{in}}, \tau_{\text{out}}$ against IoU-aware objectives instead of onset-collar F1 alone.
Cut the false-alarm rate. A tighter envelope, via per-channel or per-speaker-cluster references, plus a higher minimum-evidence rule across features, should bring the clean-audio false positives per minute down toward the $\le 1$ target.
Unlock clarity. Record paired clean and flawed takes from the same speaker and channel, which is the only valid test for mumbling, and grow the human-annotated, double-annotated test split.
Rerun all five ablations on the real dataset, since the current ablation table describes the protocol on generated tables.
Widen the corpus. Bring in the already-registered JFK inaugural and Lincoln's second inaugural, add more multi-reader texts, and give s3_fdr_infamy the readers it needs to leave permanent Norm mode.
Live coaching. Streaming alignment so a speaker sees regions appear while rehearsing, with the A/B excerpt of the skilled reading one click away.
Cross-speaker fairness audit. Evaluate the pre-registered male-to-female score-gap target ($\le 0.5$ points) on real recordings, not just the synthetic corpus.

The goal hasn't changed. A speaker should leave with a timestamp, a number and a fix, not a verdict.

Built With

cwt
docker
fastapi
histgradientboostingclassifier
librosa
mfa
numpy
opensmile
pandas
parquet
parselmouth
praat
python
pytorch
ruptures
scikit-learn
scipy
silero
streamlit

Rostrum