Inspiration## Inspiration
Evaluating competitive public speaking—such as Interpretive Reading, Declamation, Extemporaneous, and Persuasive Oratory—is notoriously subjectivespan_0span_0. Human judges must simultaneously evaluate vocal delivery (tone, cadence, volume dynamics, pause intervals) alongside textual content quality, leading to inconsistent scores across judging panelsspan_1span_1. Speakers lose critical points for delivery flaws but receive vague, unhelpful feedback like "work on your pacing" or "be more expressive" without knowing the exact millisecond they faltered or whyspan_2span_2.
While standard speech recognition datasets focus strictly on what was spoken (text transcript accuracy), there was no definitive dataset mapping high-impact rhetorical delivery against a spectrum of delivery flaws for the exact same textspan_3span_3. We were inspired to bridge this gap by treating speech as a multi-dimensional time-series signalspan_4span_4. By pairing public baseline speeches with a spectrum of intentionally injected flaws, we engineered a system that converts subjective critique into reproducible, mathematically grounded acoustic analysisspan_5span_5.
What it does
OratoMetric is an end-to-end contrastive speech analytics system that automatically detects, temporally grounds, and causally explains vocal delivery flawsspan_6span_6.
- Audio & Transcript Processing: Ingests participant speech audio alongside baseline speech dataspan_7span_7.
- Speaker-Agnostic Normalization: Transforms pitch ($F0$) and energy (RMS) into per-speaker Z-scores to eliminate biological biases (gender, native pitch range, mic volume)span_8span_8.
- Time-Series Alignment: Employs Dynamic Time Warping (DTW) and forced alignment via WhisperX to align participant timeframes with baseline timestamps regardless of pace variationsspan_9span_9.
- Temporal Flaw Grounding: Detects local acoustic deviations exceeding dynamic threshold limits ($> 1.8\sigma$) to bound exact flaw start and end timesspan_10span_10.
- Causal Explainability Engine: Translates mathematical deltas ($\Delta F0$, $\Delta \text{RMS}$, cadence distortion) into plain-English rhetorical coachingspan_11span_11.
- Interactive Dashboard: Displays a dual-waveform overlay comparing pitch/energy contours with clickable flaw regions and causal feedback cardsspan_12span_12.
How we built it
1. Data Engineering & Contrastive Spectrum
We curated prominent public speeches (TED Talks, championship debates) as pristine baselines and generated word-level alignment labels using WhisperXspan_13span_13. To build the contrastive spectrum, we programmatically injected delivery flaws into identical transcripts across a gradientspan_14span_14:
- Level 0 (Baseline): Dynamic, highly expressive public oratorspan_15span_15.
- Level 1 (Minor Flaws): Rushed cadence and awkward pause durations ($1.5\text{s} - 3.0\text{s}$)span_16span_16.
- Level 2 (Moderate Flaws): Monotone pitch contour and reduced vocal energy on key emphasis wordsspan_17span_17.
- Level 3 (Severe Flaws): Volume dropouts, erratic pacing spikes, and micro-stuttersspan_18span_18.
2. Acoustic Feature Extraction & Mathematical Normalization
Features are extracted in 25ms windows with 10ms frame hopsspan_19span_19:
- Pitch ($F0$ Contour): Extracted via Praat's autocorrelation algorithm (
Parselmouth)span_20span_20. - Vocal Energy (RMS): Calculated as: $$\text{RMS} = \sqrt{\frac{1}{N} \sum_{n=1}^{N} x[n]^2}$$
- Timbre & Vocal Clarity: 13 Mel-Frequency Cepstral Coefficients (MFCCs) and Spectral Centroidspan_21span_21.
To ensure speaker-agnostic comparison, continuous acoustic features are converted to Z-scoresspan_22span_22: $$z(t) = \frac{x(t) - \mu_x}{\sigma_x}$$
3. Dynamic Time Warping (DTW) & Flaw Bounding
We align test sequence $C$ to baseline $Q$ by finding the optimal warping path $W$ that minimizes cumulative distancespan_23span_23: $$D(i,j) = d(q_i, c_j) + \min\Big(D(i-1,j), D(i,j-1), D(i-1,j-1)\Big)$$
A Flaw Region $R = [t_{\text{start}}, t_{\text{end}}]$ is temporally grounded when the local frame distance exceeds threshold $\delta$: $$\text{Delta}(t) = \vert{}z_{\text{test}}(t) - z_{\text{baseline}}(W(t))\vert{} > 1.8\sigma$$
4. Dashboard & User Interface
Built with Streamlit and WaveSurfer.js, the frontend allows users to upload audio files, visualize normalized pitch/energy contours side-by-side, inspect highlighted flaw regions, and review causal feedback cardsspan_24span_24.
Challenges we ran into
- Non-Linear Pace Discrepancies: Participants do not speak at the exact same tempo as baseline speakers. Frame-by-frame comparison initially produced false errors. Resolving this required integrating DTW with WhisperX forced alignment to map text tokens to time-series coordinatesspan_25span_25.
- Biological Speaker Bias: Comparing male test voices against female baselines caused significant pitch delta false positives. Implementing per-speaker Z-score pitch normalization decoupled pitch height from pitch inflectionspan_26span_26.
- Background Noise Artifacts: Microphone noise distorted MFCC metrics. We integrated spectral gating noise suppression prior to feature extractionspan_27span_27.
Accomplishments that we're proud of
- Constructed a contrastive speech dataset with millisecond-accurate alignment labelsspan_28span_28.
- Engineered a robust, speaker-agnostic time-series comparative engine using DTW and Z-score normalizationspan_29span_29.
- Developed a causal explainability model that connects raw signal deltas ($\Delta \text{RMS}$, $\Delta F0$) directly to human-readable speech coachingspan_30span_30.
- Created a functional interactive web dashboard featuring synchronized dual-waveform time-series overlaysspan_31span_31.
What we learned
We learned that speech delivery quality can be quantified objectively when pitch height and overall speed are decoupled from relative dynamics and intonationspan_32span_32. Non-linear time-series alignment (DTW) provides the structural foundation required for AI models to deliver precise, grounded vocal coaching rather than generic feedbackspan_33span_33.
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for OratoMetric: Contrastive Speech Analytics & Flaw Grounding
No AI is used in writing truly from dev .
Made with Google studio
Log in or sign up for Devpost to join the conversation.