Vocalis

Contrastive speech evaluation powered by WhisperX alignment and speaker-normalized F0/energy contours to pinpoint delivery flaws, explain acoustic deltas, and generate reproducible rubric scores.

Comment

Inspiration

Judging competitive speaking—like debate, declamation, or persuasive oratory—is notoriously subjective. Because human judges weigh tone, cadence, and text differently, speakers often lose points without getting clear feedback on where or how their delivery slipped.

When we looked at existing AI speech tools, we noticed two big problems: most are just LLM wrappers over text transcripts that are completely blind to how you actually sound, while acoustic tools use rigid pitch or WPM cutoffs that penalize speakers just for having a deeper voice or a quieter microphone. On top of that, there was no clean benchmark dataset pairing great public speeches against controlled, flawed versions of the exact same text. We built Vocalis to fix that.

What it does

Vocalis is a contrastive speech analytics studio that compares a speaker's recording word-by-word against a reference baseline after normalizing for each speaker's natural vocal profile.

  • Pinpoints 6 Delivery Flaws: Grounds the exact [start, end] timestamps for rushed Pace, Slow dragging, awkward mid-clause Pauses, Monotone pitch, Volume Drops, and word Stumbles.
  • Works on Any Speech: Supports benchmark comparisons (JFK, Eisenhower, FDR, Lincoln), custom paired recordings, or Universal Mode—which auto-transcribes any uploaded speech with WhisperX and synthesizes an ideal reference baseline on the fly.
  • Deterministic Rubric & Telemetry: Generates a reproducible 0–100 score across four rubric dimensions alongside frame-level signal telemetry: FFT Spectral Centroid, 13-MFCC Similarity, Vocal Clarity (Praat HNR), WPM, and Acoustic Stress Peaks.
  • Interactive Studio UI: Lets users click any flagged region or word token to seek audio instantly, audition the reference speaker's phrasing side-by-side, and inspect time-warped pitch and energy contours.

How we built it

  • Hybrid Contrastive Dataset: We paired 4 public-domain speeches ($371\text{ s}$ total) with 72 programmatically perturbed clips (6 flaws $\times$ 3 severities) plus 4 real human validation recordings. Instead of standard phase-vocoder stretching—which leaves metallic artifacts a detector could cheat on—we used deterministically seeded Praat PSOLA resynthesis (parselmouth) for clean, pitch-preserving time and pitch edits.
  • Speaker-Agnostic Feature Extraction: Audio is resampled to $16\text{ kHz}$ mono ($10\text{ ms}$ frame step). We run two-pass Praat $F_0$ tracking to prevent octave jumps and convert Hz into semitones relative to the speaker's median, putting male and female voices on an identical logarithmic scale: $$s(t) = 12 \log_2 \left(\frac{F_0(t)}{\operatorname{median}t(F_0)}\right)$$ We normalize RMS energy against the speaker's 90th-percentile voiced level ($E{\text{rel}}(t) = E_{\text{dB}}(t) - L_{90}$) to eliminate microphone gain differences, and extract CMVN-normalized 13-MFCCs and Harmonics-to-Noise Ratio (HNR).
  • Contrastive Detection & Rubric Math: Across 5-word sliding windows $W_i$ over aligned words, we compare local duration ratios ($r_{\text{pace}}$), $\operatorname{P90}-\operatorname{P10}$ pitch spread ratios ($r_{\text{pitch}}$), loudness deltas ($\Delta L$), and punctuation-aware pause gaps: $$r_{\text{pace}}(i) = \frac{\sum_{j \in W_i} d_{\text{base}}(j)}{\sum_{j \in W_i} d_{\text{part}}(j)}, \quad r_{\text{pitch}}(i) = \frac{\operatorname{P90}(s_{\text{part}}, W_i) - \operatorname{P10}(s_{\text{part}}, W_i)}{\operatorname{P90}(s_{\text{base}}, W_i) - \operatorname{P10}(s_{\text{base}}, W_i)}$$ Regions shorter than $2.5\text{ s}$ of baseline span are filtered out to prevent jitter, and rubric penalties saturate hyperbolically ($\text{Penalty}_{\text{eff}} = \frac{85 P}{P + 85}$) so multi-flaw takes degrade smoothly.

Challenges we ran into

  • WhisperX Swallowing Pauses: When we injected a $2.5\text{ s}$ pause between "man" and "holds", WhisperX (wav2vec2) absorbed the entire silence into the timestamp of "holds". Naive gap detection saw a $0.02\text{ s}$ gap—missing the pause completely and triggering 148 false-positive slow-speech flags. We solved this by building a Silence-Aware Word Boundary Refiner that scans each word's internal energy contour against an adaptive noise floor and trims internal silences $\ge 0.20\text{ s}$: $$\theta_{\text{sil}} = \min\left(\max(L_{\text{noise}} + 10\text{ dB},\; L_{90} - 30\text{ dB}),\; L_{90} - 15\text{ dB}\right)$$
  • Long-Form Audio Compression: When testing a full 12-minute FDR speech, single-word overlap matched our 89-second FDR benchmark excerpt and crammed all word alignments into the first 31 seconds. Adding duration ratio guards, ordered bigram matching (SequenceMatcher), and 25-second VAD chunking made long-form alignment seamless.

Accomplishments that we're proud of

  • 97.3% Fewer False Positives: Our silence-refinement step dropped synthetic false positives from 148 ($1.32\text{/min}$) down to 4 ($0.04\text{/min}$) while boosting pause recall to 100% ($0.95$ mean IoU).
  • Strong Cross-Speaker Accuracy: Reached 85% overall recall across all 72 clips (98% on moderate/severe flaws) and just $0.80\text{ FP/min}$ on an unseen human validation recording.
  • 77 Passing Tests & Clean Studio UX: Verified 72/72 dataset clips automatically and built a custom carbon-zinc web workstation (FastAPI, WaveSurfer.js, Plotly.js) with zero template bloat.

What we learned

  • Combine Neural Alignment with Classical DSP: Neural forced aligners are great at finding phonetic centers of gravity, but classical energy thresholding is crucial for exact millisecond onset/offset boundaries.
  • Knowing When Not to Tune: We built a 2-fold cross-speech grid search to optimize our detector thresholds, only to find that tuned thresholds scored worse on held-out speeches ($0.79$ F1) than our untuned domain defaults ($0.89$ F1) due to speaker-cadence overfitting. Keeping the untuned defaults was a great lesson in strict validation hygiene.

What's next for Vocalis

  • Live Streaming Mode: Moving our sliding-window detector to WebSockets for real-time pacing and pitch feedback during live practice.
  • Sub-Word Articulation & Disfluencies: Adding phoneme-level posteriorgrams to catch slurred consonants and filler vocalizations ("um", "uh").
  • Multilingual & Tournament Support: Expanding reference baselines to syllable-timed languages and adding cohort analytics for debate and forensics teams.

Built With

Share this project:

Updates

Submission history