Contrastive_Speech_Analytics_and_Temporal_Flaw_Grounding

Live SpeechFlaw dashboard: synchronized waveforms, pitch contours, and millisecond flaw grounding.

Inspiration

Most public speaking and presentation feedback is frustratingly subjective: "sound more confident," "watch your pacing," or "improve your tone."

Modern generative AI coaching apps have a fatal blindspot: they convert audio into text and run a language model over the transcript. In doing so, they throw away 100% of the acoustic truth—failing to detect that a speaker dropped 4 semitones of pitch inflection at second 7.2 or dragged their speaking cadence to half-speed during their core argument.

We built SpeechFlow to turn oratorical evaluation from subjective guesswork into an objective, data-driven science. By contrastively pairing participant audio against gold-standard master speeches (such as Steve Jobs, MLK, JFK, and Winston Churchill), our engine temporally isolates delivery flaws down to the exact millisecond, produces mathematically grounded causal explanations, and delivers reproducible 0–100 rubric scores with zero LLM hallucinations.

What it does

SpeechFlow is a full-stack automated speech evaluation platform:

Universal Audio Ingestion: Accepts .m4a (iPhone Voice Memos), .aac, .mp3, and .wav files, automatically transcoding them to 16 kHz 16-bit mono streams.
Dynamic Time Warping (DTW) Registration: Computes 13-dimensional MFCC feature vectors and dynamically warps participant speech against reference master baselines to resolve non-linear tempo variations.
Cadence Slope Derivative: Analyzes the DTW warping path slope ($\frac{db}{dp}$) to mathematically detect rushed cadence ($\ge 1.25$) and dragged pauses ($\le 0.78$).
Millisecond Temporal Flaw Grounding: Detects, classifies, and bounds flaw intervals ($[t_{\text{start}}, t_{\text{end}}]$) across pacing, pauses, pitch variation, vocal energy, and clarity.
Deterministic Causal Coaching: Delivers actionable, un-hallucinated coaching drills based on exact mathematical deltas rather than black-box prompts.
5-Metric Weighted Rubric: Computes weighted rubric scores across Pacing (25%), Pauses (20%), Pitch Variation (20%), Energy Projection (15%), and Articulation Clarity (20%).
Interactive Multi-Track Dashboard: Features synchronized waveforms with colored flaw overlays, synchronized Plotly pitch/energy contours, and click-to-seek flaw review.

How we built it

Signal Processing & ML Core: Python 3.11, Librosa, SciPy, NumPy, SoundFile, ImageIO-FFmpeg. We implemented pYIN fundamental frequency extraction, $O(N \log N)$ FFT-based autocorrelation for Harmonic-to-Noise Ratio (HNR), local jitter/shimmer estimation, and acoustic Voice Activity Detection (VAD).
Backend API: FastAPI and Uvicorn structured with asynchronous endpoints, Dockerized, and deployed 24/7 on Render.com with startup baseline pre-warming.
Frontend Dashboard: Built with React 18, Vite, and Tailwind CSS. We integrated WaveSurfer.js (with timeline and regions plugins) alongside multi-track Plotly.js charts for continuous contour scrubbing. Deployed globally on Vercel.
Empirical Benchmarking: Engineered a calibrated dataset ($L_1$ to $L_5$ degraded deliveries evaluated against $L_6$ master baselines) with 25 automated unit and calibration tests run via Pytest.

Challenges we ran into

Universal Codec Ingestion: iPhone voice memos use .m4a / AAC containers, which frequently crash lightweight Linux Docker images lacking proprietary decoders. We built a universal transcoding pipeline with FFmpeg that reliably normalizes any audio container into clean mono PCM.
Variable Oratorical Cadence: Different speakers deliver words at vastly different speeds. Linear time stretching distorted natural speech patterns. We solved this by developing non-linear Dynamic Time Warping across 13 MFCC dimensions, utilizing the cadence derivative $\frac{db}{dp}$ to isolate local tempo fluctuations.
Cloud Performance on Constrained Compute: Running frame-by-frame $O(N^2)$ cross-correlations across 2,600 frames on Render’s 0.1 CPU free tier took over 3 minutes. We redesigned the engine using Fast Fourier Transform autocorrelation and 8 kHz pitch downsampling, slashing computation time by 12x down to ~10–15 seconds.
Zero-Hallucination Constraint: Ensuring every coaching tip is strictly derived from verified mathematical deviations rather than generative text models.

Accomplishments that we're proud of

Spearman Rank Correlation $\rho = 0.8542$: Validated across our benchmark dataset, proving monotonic rubric degradation as delivery flaws increase.
High Temporal Precision: Attained a Mean IoU of $0.7169$ and $79.6\%$ precision on millisecond flaw interval grounding.
100% Test Coverage: All 25 out of 25 unit, acoustic, and calibration tests pass.
Production Cloud Deployment: A seamless, live full-stack system linking a high-performance React UI on Vercel to a 24/7 containerized FastAPI AI backend on Render.

What we learned

The mathematical nuances of non-linear DTW alignment and acoustic feature extraction.
How to replace generative AI guesswork with deterministic signal processing to achieve transparent, auditable AI evaluation.
Containerization best practices, low-latency audio processing, and optimizing heavy ML workloads for constrained cloud environments.

What's next for SpeechFlaw

Real-Time WebSocket Streaming: Live speech analysis during practice runs with sub-second visual feedback.
Multimodal Video Grounding: Incorporating computer vision to analyze eye contact, facial tension, and body posture synchronized with vocal contours.
Personal Voice Profile Adaptation: Enabling users to record their own baseline profile and track oratorical growth over time.