-
-
SpeechFlaw: pinpoints where a speech delivery goes wrong, down to the timestamp, and explains why in acoustic terms.
-
Streamlit dashboard: rubric scores for pace, intonation, volume, clarity and pausing, with both recordings side by side.
-
Shaded bands are detected flaws; dashed boxes are the true injected regions. Grey = ideal (DTW-aligned), blue = participant.
-
Five aligned views: pitch, pitch variability, energy, local speech rate (DTW slope) and >2 kHz clarity, all speaker-normalised.
-
Each flaw states the measured value, the baseline/threshold and the perceptual effect, with a player that jumps to that moment.
-
Stress test L1: one small extra pause was injected, and it is the only flaw reported. Overall score 9.6/10.
-
Stress test L5: rushing, monotone, trailing off, mumbling, dragging and broken pauses, all found and timestamped. Score 4.4/10.
-
Overall score falls with flaw severity across the contrastive spectrum, from 10.0 (ideal vs itself) to 4.4 (botched).
-
Contrastive dataset: flaws injected with signal-processing operators, so every label has exact ground-truth timestamps.
-
Pipeline: 10 ms features (Praat F0, energy, MFCCs), speaker normalisation, MFCC-DTW alignment, then thresholded, explained flaws.
-
Evaluation vs ground truth: 95% recall, 91% precision, 0.1-0.4 s typical onset error, per-flaw recall and IoU.
-
Upload mode: bring your own ideal and participant recordings (wav, mp3, m4a) plus an optional transcript for the same analysis.
Inspiration
What it does
SpeechFlaw analyses a participant’s speech delivery performance versus a perfect version of the same text. It matches the two audio files by means of Dynamic Time Warping, identifies exactly when each delivery flaw starts and ends (rushing, dragging, monotone, trailing off, mumbling, gaps in pausing), describes each flaw with acoustical values, and evaluates the performance according to a scoring rubric.
How we built it
Contrastive dataset: from one perfect reading I create 11 flawed versions through the signal processing algorithms (phase-vocoder time stretch, Praat PSOLA pitch flatness, low-pass filtering, gain reduction and added silence). Since the flaws are introduced by me, the ground-truth timestamps are known for each label. Features: F0, energy, spectral centroid, >2 kHz energy ratio, pauses and MFCCs are extracted in 10ms frames. Normalization makes the system analyze only the delivery, not voice. Alignment: MFCC-based DTW algorithm aligns every participant frame with the same word in the perfect reading. Per-frame differences are thresholder and labelled with flaw descriptions and causative parameters. Dashboard: Streamlit app visualizes time series, highlight flaw regions and scores according to rubric. User can upload their own audio.
Challenges we ran into
There was no existing dataset. There’s no contrastive dataset pairing good and bad readings of the same text, so I couldn’t just grab some data and train. I had to create a contrastive dataset from scratch, introducing faults using signal processing operators so that each label has the start and end timestamps. Flaws in pitch tracking obscured monotone speech. The first monotone detector detected zero out of four flats. The pitch tracking algorithms do jumps across octaves in voicing boundaries, and these outliers affected the standard deviation and masked the actual flat speech. Using the percentile spread ((P90 − P10) / 2.563) instead of the regular spread raised the monotone detection rate from 0% to 100%. Pauses appeared to be slow speech. The insertion of a pause makes the DTW alignment longer and caused the pacing detector to see it as a “dragging” reading. This flaw was corrected by calculating the speed based on speech frames only. Coupled Features. Time-stretching speech reduces the rate of change of the pitch, so that a dragging reading can partially trigger the monotone detector. That’s the last remaining false positive that I reported. Time. I did everything by myself: designed, created, tested and documented.
Accomplishments that we're proud of
95% recall and 91% precision compared to exact ground truth, with flaw timestamps always between 0.1–0.4 s. No false positives in ideal recording – it is compared with itself and thus has 0 flaws, yielding perfect 10. Scores along the severity spectrum: 10.0 → 9.6 → 9.9 → 7.3 → 5.8 → 4.4 from ideal to messed up. Explanations with concrete numbers: all flaws include the observed value, the baseline and its meaning, e.g. “DTW warp slope 1.48 → speaking 48% faster than ideal”. Deterministic: no randomness in inference at all, and python cli.py eval yields exactly the same score every time. Speaker agnostic by construction: pitch and energy are normalised to speaker-specific baselines, so the system evaluates speaking style, not the voice type.
What we learned
Core concepts from signal processing and system theory (FFTs, spectral energy bands, filters, pitch detection via autocorrelation) apply directly to a practical task such as evaluating speech. The Dynamic Time Warping algorithm is robust; once two recordings are synchronized, all comparisons become just one-to-one comparisons of individual frames. Robust statistical methods are essential for real signals. Just one outlier-sensitive statistic proved sufficient to ruin an entire detector. Truth labeling is more important than a complicated model. Precise labels enabled us to evaluate if each modification made a difference. Honest discussion of limitations strengthens, not weakens, a project.
What's next for SpeechFlaw
Actual human errors: record actual human beings actually rushing, speaking unclearly and losing their inflection and verify the detectors using actual human errors. Word-level stressing and emphasizing: see if the speaker stresses the right words and not if he or she changes her pitch levels. Detecting filler words: detect "um", "uh" and repeated words based on the transcript alignment. Machine learning models: create a classifier using this dataset to generalize across speakers, rooms and microphones. Practice mode: real time feedback to a student during practice for debate clubs and elocution contests. Support for Indian languages: extend to Hindi and Kannada speeches since no component of the acoustic pipeline depends on English.
Log in or sign up for Devpost to join the conversation.