Inspiration
Public speaking, courtroom litigation, broadcast journalism, and high-stakes technical interviews shape careers, decisions, and leadership. Yet oratorical evaluation has historically remained trapped in the dark ages of subjective human judging—rife with cognitive fatigue, subconscious bias, and vague platitudes like "just speak with more confidence" or "work on your pacing."
When speakers turn to modern "AI speech coaches," they encounter a subtle deception: virtually every commercial speech app is merely a thin LLM wrapper around a generic speech-to-text transcript. They convert audio into text tokens, discard the physical acoustic waveform, and prompt a generative model to hallucinate arbitrary confidence numbers (e.g., "Confidence: 78%"). An LLM looking at text cannot tell you that your pitch dropped by 4.2 semitones, that your voice attenuated by 14\text{ dBFS}, or that an unnatural pause fractured the cadence of your argument.
We built Speech Guru to solve this foundational flaw. We treat human speech as what it physically is: a continuous multimodal time-series signal. Inspired by the Track C challenge, we engineered an end-to-end platform that delivers millisecond-accurate temporal flaw grounding, mathematical acoustic evidence, and interactive practice workspaces—with zero generative LLM hallucinations in the acoustic scoring path.
What it does
Speech Guru transforms oratorical training from subjective guessing into an empirical acoustic science:
- Millisecond-Accurate Temporal Flaw Grounding Rather than assigning a single arbitrary score to an entire speech, Speech Guru pinpoints the exact temporal intervals [t_{\text{start}}, t_{\text{end}}] where delivery falters across 7 core acoustic flaw categories:
- Pacing Deviations: Runaway rushes (TOO_FAST > 190\text{ WPM}) and sluggish drags (TOO_SLOW < 110\text{ WPM}).
- Vocal Projection Deviations: Under-projected whispers (LOW_ENERGY < -26\text{ dBFS RMS}) and clipping overdrive (HIGH_ENERGY > -12\text{ dBFS RMS}).
- Melodic Inflection: Robotic delivery (MONOTONE with pitch variance \sigma(F_0) < 5\text{ Hz}).
- Dysfluent Pausing: Awkward intra-sentence silence gaps (EXCESSIVE_PAUSE > 800\text{ ms}).
- Terminal Rising Intonation: Inappropriate up-talk questioning inflections on declarative statements.
- Causal Acoustic Evidence & Verifiable Truth Every flaw is backed by concrete mathematical deltas comparing participant delivery against reference physics: \Delta F_0 = 12 \log_2 \left( \frac{f_{\text{speaker}}}{f_{\text{ref}}} \right) \quad [\text{semitones}] \text{dBFS} = 20 \log_{10} \left( \frac{\text{RMS}(s)}{\text{RMS}{\text{max}}} \right) \text{Local Cadence } \rho = \frac{N{\text{words}}}{t_{\text{end}} - t_{\text{start}}} \times 60 \quad [\text{WPM}]
Users receive deterministic, actionable diagnosis cards (e.g., "Pacing accelerated by 38% across words 14–19; pitch dropped 4.2 ST"), completely eliminating generative AI guesswork.
5 Purpose-Built Practice Workspaces a) Speech Assessment Sandbox: Dual-waveform synchronizer with real-time audio scrubbing, synchronized transcript highlighting, and toggleable F_0 pitch track and RMS energy curve overlays. b) Industry AI Interviewer: Turn-based technical and behavioral mock interview simulations with automated STAR framework adherence checks, filler-word counters, and deterministic keyword-triggered follow-up probes across 20+ career domains. c) News Anchor Studio: Professional broadcast studio teleprompter with auto-scroll speed controls, dynamic WPM pacing indicators, and real-time wire copy. d) Public Speaking & Debate Arena: Parliamentary debate sparring with opposing argument generation, realistic ambient crowd soundscapes, and an integrated 4-7-8 tactical breathing pacer for stage fright regulation. e) Storyteller Studio: Expressive narration coaching guided by Freytag's Dramatic Pyramid, tracking narrative tension build-up and vocal acoustic climax.
"You vs. You" & 1v1 Asynchronous Arena
Track personal longitudinal progress across attempts or challenge peers in head-to-head oratorical battles scored with speaker-invariant dual-channel normalization.
How we built it
Speech Guru is engineered upon a Decoupled Two-Pillar Architecture separating conversational intelligence from acoustic physics:
Pillar B: Acoustic Measurement & Temporal Grounding Engine (Pure Math & DSP) Audio Ingestion & Forced Alignment: PyAV and FFmpeg decode incoming audio (WAV, MP3, WebM/Opus) into 16 kHz mono float32 arrays. Forced alignment via torchaudio.pipelines.MMS_FA anchors speech to exact word-level time tokens.
Contrastive Two-Channel Normalization: Solves biological vocal tract differences (e.g., baritone vs. soprano) without erasing actual delivery flaws: Magnitude Channel: Preserves absolute dynamic deviations via semitone (\Delta \text{ST}) and relative decibel (\Delta \text{dBFS}) deltas.
Shape Channel: Z-score contour normalization (z_t = \frac{x_t - \mu}{\sigma}) isolates tempo and melodic contour shapes independently of vocal pitch baseline.
Hybrid Neural & DSP Detection: Merges classical acoustic phonetic extraction (Praat Parselmouth autocorrelation F_0, RMS sound energy, spectral centroids) with a frozen facebook/wav2vec2-base encoder and a multi-label classification head trained on our Tier 1 dataset.
Deterministic Explanation Generator: Strongly typed Jinja2 templates consume acoustic deltas to produce human-readable diagnostic cards without contacting any external LLM.
Pillar A: Conversational & Workspace Intelligence Layer Deterministic BM25 RAG Knowledge Engine: Catalogs 20+ specialized industry question banks, AP wire broadcast copy, and parliamentary debate motions, retrieving targeted follow-up prompts and sparring cases deterministically.
Full-Stack Production Infrastructure Backend: FastAPI (Python 3.11), Uvicorn, and MongoDB for TTL-indexed session persistence, leaderboard rankings, and NIST 800-63B authentication with stdlib scrypt password hashing. Frontend: React 19, TypeScript, and Vite designed under a sleek "Minimal Editorial" aesthetic with seekable waveform playback, uPlot interactive contours, and zero third-party font/CDN leaks.
Custom Dataset & Training 1,680 Physical Audio Files: Scaled real-world speech dataset derived from 40 human speakers in OpenSLR LibriSpeech dev-clean. Contrastive Flaw Balance: 240 clean baseline seeds perturbed across 6 physical flaw operators (phase-vocoder time-stretch, gain attenuation/boost, median F_0 flattening, silence insertion) yielding a 1:7 contrastive balance (240 files per class across all 7 categories). Strict Speaker-Disjoint Splits: 80% Train (32 speakers, 1,344 samples) / 20% Validation (8 completely unseen speakers, 336 samples).
Training Results: Trained for 35 epochs on NVIDIA GPU; achieved record low validation loss of 0.1357 at Epoch 24 with Event F1 score > 0.90 under acoustic stress testing (noise down to 10 dB SNR, reverb up to RT60 = 0.6s).
Challenges we ran into
a) The 365-Second Inference Bottleneck (24x Acceleration): Initial pipeline runs on 120-second speech recordings took over 360 seconds due to repetitive frame extraction loops and disk-bound subprocesses. We eliminated this bottleneck by vectorizing frame feature extraction into a single pass, batching PyTorch tensor operations (\text{batch_size}=16), and streaming in-memory chunked Whisper audio arrays, slashing runtime to 11.9 seconds (a 24x speedup).
b) The Normalization Paradox: Standard global z-score normalization mathematically eliminates monotone flaws (by forcing standard deviation to 1.0) and erases low-energy flaws. We solved this with our novel Two-Channel Normalization: preserving magnitude deltas in physical units (semitones / dB) while standardizing contour shape for speaker-invariant cadence matching.
c) Cross-Browser Audio Decoding & Libsndfile Crashes: Browser MediaRecorder outputs containerized WebM/Opus packets that crash standard C-based audio libraries (libsndfile). We engineered PyAV stream decoders with an in-memory seekable WAV fast-path, guaranteeing flawless playback and analysis across Chrome, Edge, Safari, and Firefox.
d) Speaker Identity Leakage in Speech ML: Random audio chunk splitting leads to neural networks memorizing speaker timbre rather than detecting vocal delivery flaws. We enforced strict speaker-disjoint splits, ensuring that validation speakers never appeared in the training corpus.
Accomplishments that we're proud of
a) Zero Generative LLM Hallucinations in Scoring: Every score (0–100) and feedback card is 100% deterministic, reproducible bit-for-bit, and backed by verifiable physical acoustic data.
b) Instant Millisecond Waveform Seeking: Clicking any flaw card in the UI immediately seeks the waveform to the exact syllable and displays the acoustic delta.
c) 24x Inference Speedup: Reduced processing time on 2-minute speech files from 365 seconds to 11.9 seconds without sacrificing mathematical precision.
d) 1,680 Cleanly Balanced Real Audio Files: A fully documented, open, reproducible contrastive speech dataset with strict speaker-disjoint validation and rights manifests.
e) Enterprise Security & Privacy-by-Design: NIST 800-63B compliant authentication, zero raw audio persisted on servers, MongoDB TTL auto-cleanup, and complete zero-CDN client-side security.
f) 95 Automated Tests Passing: 100% deterministic reproducibility across API routes, scoring contracts, and RAG catalog retrievers.
What we learned
Acoustic DSP beats LLM prompts every single time for oratorical evaluation. Self-supervised speech representations (wav2vec 2.0) combined with classical acoustic phonetics (Praat) provide an empirical grounding that no text LLM can replicate.
Vectorization and memory layout are critical when processing multi-minute multimodal audio streams in production web services.
Ethical AI in Speech: Designing evaluation tools that judge delivery mechanics rather than accent, timbre, or demographic identity requires meticulous contrastive normalization and speaker-disjoint validation.
What's next for Speech Guru
a) Real-Time Streaming Grounding via WebSockets: Delivering sub-200ms instantaneous acoustic feedback directly into headphones as speakers talk.
b) Multilingual Contrastive Models: Expanding MMS_FA forced alignment to support Spanish, French, German, Hindi, and Mandarin.
c) Multimodal Video & Computer Vision Fusion: Fusing acoustic energy and pitch contours with webcam computer vision to analyze eye contact, facial tension, and posture gestures in real time.
d) Live Classroom & Tournament Mode: Real-time synchronous judging dashboards for collegiate debate tournaments and public speaking tournaments.
Log in or sign up for Devpost to join the conversation.