Inspiration
Public speaking and communication coaching tools frequently promise to make speakers more confident and effective, yet existing solutions suffer from two fatal flaws:
- Unexplainable Black-Box Scoring: Users receive an ambiguous single-number score (e.g., "78% confidence") without knowing which acoustic behaviors caused the penalty or where they occurred.
- LLM Timestamp Hallucination: Generic AI speech wrappers ask large language models to "find awkward pauses and fillers with timestamps," resulting in invented time intervals that bear zero connection to physical audio waveforms.
Speakers don't need vague compliments or fabricated timestamps. They need to know: What happened? Where did it happen? What physical evidence proves it? And how do I fix it?
This motivated SpeechProof—an evidence-grounded speech evaluation platform that replaces subjective impressions with physical acoustic signal processing, sub-second speech-to-text alignment, deterministic rubric scoring, and contrastive matched-pair analytics.
What It Does
SpeechProof's core promise:
“Don’t just receive a speech score. Discover what happened, where it happened, what evidence supports it, and how to improve.”
- Temporal Flaw Grounding: Detects prolonged awkward silences ($>1.4$s), verbal fillers ("um", "like", "you know"), rushed cadence ($>165$ WPM), monotone pitch ($F_0\text{ Std} < 16$ Hz), and volume drop-offs, emitting each finding as a verified evidence object with millisecond timestamps.
- Clickable Audio Interval Playback: In the interactive dashboard, users can click any detected flaw to listen directly to the exact $300$ms audio slice where the issue occurred.
- Deterministic 0–100 Rubric Engine: Computes five transparent delivery dimensions—Pace (20%), Fluency (20%), Pausing (20%), Vocal Variation (20%), and Volume Consistency (20%)—using documented mathematical equations.
- Contrastive Matched-Pair Analytics: Directly compares ideal vs. flawed recordings of identical speech scripts side-by-side, verifying that controlled flaws produce targeted metric and score shifts.
- Actionable Rehearsal Drills: Provides step-by-step physical practice drills (such as "The 2-Second Silent Inhale Drill" and "Bridge Transitions") linked directly to observed defect intervals.
- 100% On-Device & Zero API Keys: Runs locally in-process on CPU via
faster-whisperandlibrosa. Zero cloud API dependencies; zero audio data leaves your machine.
How We Built It
SpeechProof is structured as a modular, high-performance Python monolith:
1. Dual-Stream Acoustic & Lexical Pipeline
- Acoustic Signal Engine: Uses
librosa,soundfile, andscipyto extract frame-by-frame Active-Speech RMS Dynamics (loudness restricted strictly to speech frames within $35$ dB of peak), YIN Fundamental Frequency ($F_0$) Tracking (pitch modulation across $65$–$350$ Hz), and energy-based silence segmentation. - Local Speech Recognition: An in-process
faster-whispermodel (tiny.en, int8 quantized on CPU) generates segment-level and word-level timestamps, lexical filler word occurrences, and cadence measurements.
2. Mathematical Rubric Formulation
Each category maps deterministically between 0 and 100:
- Pace (20%): Target window: $125$–$165$ WPM. $$\text{Score}_{\text{pace}} = \max\left(15, 100 - \frac{|\text{WPM} - \text{Target}|}{35} \times 60\right)$$
- Fluency (20%): Target: $\le 1.0$ fillers/min. Degrades linearly from 100 down to 40 at 6 fillers/min.
- Pausing (20%): Ideal pause ratio between $12\%$ and $28\%$, combined with quadratic penalties for awkward silences $>1.4$s.
- Vocal Variation (20%): Evaluates dynamic pitch inflection ($F_0\text{ Std}$ between $20$ Hz and $55$ Hz), heavily penalizing flat monotone delivery ($F_0\text{ Std} < 16$ Hz).
- Volume Consistency (20%): Measures active-speech RMS standard deviation ($2.0$–$7.5$ dB optimal), penalizing microphone fading or trailing off ($>9.5$ dB).
3. Interactive Web Dashboard
Built with Streamlit and Plotly featuring an executive dark research theme, dynamic Gantt flaw timelines, interactive loudness/pitch curves, and downloadable JSON/CSV reports.
Contrastive Dataset & Empirical Benchmark
To evaluate the system scientifically, we curated SpeechProof-Contrastive-v1—a controlled benchmark of 5 matched pairs (10 WAV audio recordings, 134.6 seconds total) with ground truth annotations in data/manifest.csv:
- P01: Elevator Pitch — Ideal vs. Excessive Fillers ("um", "like", "you know")
- P02: Cloud Architecture Keynote — Ideal vs. Prolonged Awkward Silences ($>3.2$s dead air)
- P03: Biometric Sensor Launch — Ideal vs. Hyper-Rushed Pacing ($>265$ WPM)
- P04: Executive Strategy Briefing — Ideal vs. Monotone Robotic Delivery ($F_0\text{ Std} = 0$ Hz)
- P05: Customer Advisory Postmortem — Ideal vs. Diaphragmatic Volume Drops ($-22$ dB fade)
Empirical Results:
- Flaw Grounding Accuracy: 100.0% (5 / 5 True Positives)
- Precision / Recall / F1: 1.000 / 1.000 / 1.000 (Zero False Positives)
- Mean Score Separation: -15.5 points delta between ideal and flawed deliveries
- Execution Speed: Full 10-clip benchmark runs in under 30 seconds on standard CPU.
Challenges We Ran Into
- The Silence Energy Skew: Computing global RMS standard deviation across an entire audio file caused standard deviations of $35+$ dB purely due to natural sentence pauses. We solved this by developing an active-speech frame filtering algorithm that isolates spoken vocal projection from room silence.
- Sub-Second Audio Slicing: Connecting Streamlit UI buttons to immediate playback of specific flaw intervals required building an in-memory byte-buffer WAV slicer with edge padding to avoid audible clicking artifacts.
- Zero API Key Requirement: Ensuring the entire pipeline runs locally on standard CPU without expensive GPUs or cloud APIs required optimizing
faster-whisperwith int8 quantization and singleton model caching.
Accomplishments That We're Proud Of
- Achieving 100% flaw grounding accuracy with zero false positives on our contrastive benchmark.
- Eliminating LLM hallucination in timestamp generation by enforcing physical acoustic signal grounding.
- Passing an automated 11-test
pytestregression suite in under 3 seconds. - Delivering a turnkey application ready for judges to launch in a single command.
What We Learned
- Grounded multimodal AI requires tight coupling between acoustic signal processing (continuous domain) and lexical token alignment (discrete domain).
- Matched contrastive pairs are essential for speech evaluation research; comparing completely different speakers on different topics introduces too many confounding variables to validate algorithmic accuracy.
What's Next for SpeechProof
- Neural VAD Integration: Adding Silero VAD for enhanced voice-activity detection in noisy environments.
- Multilingual Filler Dictionaries: Expanding beyond English to Spanish, Hindi, French, and German.
- Phoneme-Level Articulation: Integrating Wav2Vec2 CTC forced alignment for fine-grained pronunciation feedback.
Log in or sign up for Devpost to join the conversation.