detection vs severity graph
benchmark explorer
Flaw Studio
method
Home Page
Results
main analysis screen

Inspiration

In debate, declamation and public speaking, judges often say "your delivery was weak", but not where or why. Three judges can score the same speech three different ways. We wanted feedback that is exact, explainable and reproducible, and that lets you hear what "better" sounds like in your own voice.

What it does

SHRUTI compares your delivery of a famous speech with good deliveries of the same text, word by word. It finds 10 kinds of delivery flaw:

rushed or dragged phrases
long pauses, and missing pauses
monotone stretches
a fading voice
mumbling
fillers ("umm"), stutters
misplaced emphasis

For every flaw it:

pins the exact start and end time, with a bootstrap 95% interval;
proves it with a statistical test under false-discovery-rate control;
explains it with the measured vs expected numbers, the maths and a fix;
plays back your own voice with only that flaw corrected (Ghost Coach), then re-analyses the corrected audio to confirm the cause.

Drill mode lets you re-record one phrase and watch the score change. Scores cover 7 dimensions (0–10) plus a WSDC-style Style mark.

How we built it

Dataset (SHRUTI-Bench). No public dataset pairs good and flawed deliveries of the same text, so we built one. Six public-domain US presidential speeches (JFK, FDR, Obama, Eisenhower, Reagan) form the "good" baseline. A "bad mirror" covers 10 flaw types × 10 severity levels, made with signal processing in the speaker's own voice. Clean controls, voice-changed controls and stress conditions (noise, phone, reverb, MP3) round it out. In total: 914 files, 10.5 hours, 1,160 flaws labelled to the millisecond, regenerable bit-for-bit from seeded recipes. Flaw Studio collects real human recordings, where the on-screen instruction ("rush these words") is the label.

Pipeline.

wav2vec2 CTC forced alignment, using our own Viterbi decoder.
Features every 10 ms: pitch, intensity, FFT spectral tilt and centroid, MFCC, cepstral peak prominence.
Speaker normalisation, e.g. pitch in semitones relative to your own median: $ st(t) = 12\log_2\frac{F_0(t)}{\tilde F_0} $.
A population baseline whose spread is shrunk toward a prior: $$ \tilde s^2 = \frac{\nu_0 s_0^2 + (n-1)s^2}{\nu_0 + n - 1} $$
About 450 tests per take, then Benjamini–Hochberg FDR at $ q = 0.05 $: flag every test with $ p_{(i)} \le \frac{i}{m}q $.
Regions with bootstrap confidence → deterministic explanation cards → rubric $ S_d = \max(0, 10 - \frac{60}{T}\sum c(\text{sev}) - g_d) $.
Ghost Coach re-delivers only the flawed region with TD-PSOLA.

No language model is used anywhere in scoring or explanations.

Stack: Python, FastAPI, PyTorch/transformers, Praat-Parselmouth, librosa, SciPy; React + TypeScript + Tailwind + wavesurfer.js + KaTeX.

Results (held-out speakers and texts, evaluated once)

Metric	Value
Precision	0.81
Event F1 (IoU ≥ 0.5)	0.50 (95% CI 0.43–0.55)
Median boundary error	~20 ms
False flags on flaw-free audio	0
Score vs severity (Spearman)	−0.95

Fillers (F1 0.88) and long pauses (0.78) are caught even at mild severity. Tempo, emphasis and mumbling are caught only at high severity, and we report that honestly.

Challenges we ran into

The aligner glued "umm"s and stutters onto words and dropped short words into pauses. We added global silence assignment and onset repair.
Background noise hid pauses. We added a noise-adaptive silence threshold, which cut false alarms 4×.
Raising sensitivity also raised false alarms. We kept strict FDR control so that when SHRUTI flags something, it is real.
A contrastive neural encoder we trained did not beat the statistics on an unseen speaker, so we report it as an ablation instead of shipping it.

What we learned

Precise timing matters more than a single score. Honest statistics (FDR control, held-out tests, frozen settings) build trust. And hearing your own corrected voice is far more convincing than any number.