SpeechMirror

Inspiration

Most speech-feedback tools give you one score: "78/100, speak more clearly." That doesn't tell you where you went wrong or why. A good coach plays the ideal version next to yours and says, "Here you rushed, and here you paused too long."

SpeechMirror automates that. It's a git diff for your voice: take an ideal reading of a script, take yours, and see exactly where and how they differ.

What it does

You give SpeechMirror two recordings of the same transcript, an ideal take and a participant take. It then:

Detects four flaw types: rushed pace, dead pause, flat pitch, mumbled clarity
Places each flaw on a timeline with start/end times and a severity from 0 to 4
Explains it with measured evidence, e.g. "pace +38% at 0:14–0:19"
Shows everything in an interactive dashboard: overlaid waveforms, highlighted flaw regions, click-to-play, feature timelines, rubric scores and an honest-limitations panel

How we built it

Dataset. We recorded 6 transcripts (business, science, tech, news, narrative, customer service) as ideal, rushed, dead-pause, flat-pitch and mumbled takes. That gave 28 real clips, converted to 16 kHz mono WAV with SHA-256 hashes, quality checks and a manifest. To fill out the severity scale, we built a deterministic perturbation engine (fixed seed) that injects one controlled flaw into one target sentence of an ideal take. This added 96 clips, all labelled synthetic_perturbation and kept separate from real data. Splits are by transcript, so no transcript appears in two splits.

Flaw	Parameter	Severity 1 → 4
Rushed pace	speed factor	1.2× → 1.8×
Dead pause	inserted silence	0.5 s → 2.5 s
Flat pitch	retained log-F0 variance	0.75 → 0.05
Mumbled	gain / low-pass	−3 dB, 6 kHz → −12 dB, 2.5 kHz

Features. Per frame: RMS energy, F0 (WORLD), spectral centroid, flatness, bandwidth and zero-crossing rate. Per word: duration, preceding pause, F0 spread and centroid. Everything is speaker-normalized, with F0 in semitones relative to the speaker's median and energy relative to the recording's peak, so there are no absolute-Hz or absolute-dB thresholds.

Contrast. Word timestamps come from Whisper, and participant words are compared against the ideal's on the same transcript. Over a window of $n$ words, the pace signal is

$$\Delta_{\text{pace}} = \frac{1}{n}\sum_{i=1}^{n}\left(d_i^{\text{part}} - d_i^{\text{ideal}}\right)$$

and the pitch signal compares F0 spread in semitones:

$$\Delta_{\text{pitch}} = \sigma_{F0}^{\text{part}} - \sigma_{F0}^{\text{ideal}}$$

Windows past a threshold are flagged, merged, and given a severity and a plain-language explanation.

Stack. Python, librosa, pyworld, Whisper, FastAPI, React, Docker.

Challenges

Honest evaluation was the hardest part. "Accuracy" can mean label accuracy, detection accuracy or time-grounding accuracy. We defined each separately, used Wilson confidence intervals, and audited our own benchmark for leakage and over-generous scoring.
Timing is fragile. ASR word timestamps jitter, and a long pause or a rushed sentence can shift word matching. Subtle flaws (flat pitch vs. rushed pace) get confused when timing is noisy.
One speaker. All real recordings come from a single speaker, so generalization to new voices, microphones and accents is untested.
Synthetic vs. real. Perturbations give exact ground truth, but they aren't the same as a person genuinely mumbling, so real and synthetic results are kept separate.

What we learned

A detector is only as trustworthy as its evaluation, and leakage hides in small places.
Contrastive features beat absolute ones: "compared to the ideal" is a stronger signal than "is this pitch low?"
An explanation with evidence ("pace +38%") is something a person can act on, which makes explainability the real product.

Limitations

Single speaker and a small dataset
Accuracy depends on alignment quality, and errors propagate
Sensitive to microphone and room changes
No semantic or emotional judgment, English only
Not a clinical or medical tool

What's next

Record more speakers for a true held-out generalization test
Alignment that holds up under pauses and pace changes (DTW over MFCCs)
Thresholds derived from natural ideal-vs-ideal variation to reduce false positives
PDF/JSON report export and multi-take reference envelopes