Inspiration

Most speech-coaching tools provide a single score or generic feedback such as “improve your pacing” without explaining exactly where a problem occurred or why it matters. We wanted to build something more precise: an AI-powered speech analysis system that acts like a coach, identifying delivery issues with timestamps, measurable acoustic evidence, and understandable explanations.

Inspired by iconic speeches such as John F. Kennedy’s “We Choose to Go to the Moon,” we set out to test whether a system could detect deliberately introduced flaws in speech delivery and pinpoint exactly where they occurred.

What it does

Cadence — Contrastive Speech Delivery Analytics analyzes speech recordings to identify potential delivery issues and explain them through measurable audio features.

It supports two analysis modes:

Reference Comparison: Compares a speaker’s recording with a reference delivery of the same text.
Self-Comparison: Evaluates a recording against the speaker’s own typical delivery when a reference is unavailable.

Cadence analyzes pitch variation, loudness, articulation tempo, pauses, and spectral characteristics related to clarity. It detects significant deviations, groups them into timestamped regions, estimates their severity, and generates evidence-based explanations.

An interactive Streamlit dashboard displays time-aligned comparisons, deviation-versus-threshold charts, and detailed flaw cards, helping speakers understand not only what may need improvement, but also where it happened.

How we built it

We created a dataset of 13 clean baseline recordings, including three presidential speeches, two human LibriVox readings, and eight neural text-to-speech voices. We generated 17 flawed variants per baseline, covering pacing, monotone pitch, pauses, volume drops, and reduced articulation clarity at different severity levels. This produced 221 flawed clips with time-bounded ground-truth labels.

Our pipeline combines:

Speech alignment: Whisper-based alignment techniques and dynamic time warping to estimate word-level timestamps, refined using nearby silences.
Acoustic feature extraction: Measurements of pitch, vowel-band loudness, articulation tempo, preceding pauses, and high-frequency spectral energy.
Speaker-aware normalization: Relative measurements that account for each speaker’s natural vocal characteristics.
Adaptive deviation detection: Robust noise estimation and thresholds to identify meaningful differences between a participant and a reference.
Temporal localization and severity: Consecutive deviations are grouped into time regions and assigned severity estimates.
Evaluation and visualization: Temporal intersection over union (IoU) measures detection overlap with ground-truth labels, while Streamlit visualizes the results.

We also focused on reproducibility through controlled random seeds, greedy decoding, pinned dependencies, and SHA-256 dataset manifests.

Challenges we ran into

Bridging synthetic and real speech: Thresholds calibrated on TTS recordings were too sensitive to natural variations in human voices. We added real orators and human readings to make the evaluation more representative.

Maintaining accurate timestamps: Inserted pauses shifted later word boundaries and initially caused incorrect ground-truth labels. We corrected the generation process and introduced validation checks for labelled pauses.

Ensuring deterministic results: Audio resynthesis and speech recognition could produce different outputs across runs. We controlled randomness and decoding settings to improve reproducibility.

Avoiding misleading features: Boundary jitter and spectral flux sometimes confused pacing changes with other delivery issues. We refined our articulation-time measurements and selected a clarity-related feature based on its ability to distinguish slurring from other flaws.

Handling real-world recordings: Applause and background sounds complicated the analysis of public speeches. We evaluated temporal overlap over speech regions only.

Comparing different speakers: Natural differences in voice and delivery can be as large as moderate injected flaws. Reliable cross-speaker comparison remains one of our most difficult problems.

Accomplishments that we're proud of

Built an end-to-end speech analysis pipeline that moves beyond a single overall score to timestamped, interpretable feedback.
Created 221 flawed speech clips with time-bounded ground-truth labels for systematic evaluation.
Achieved a mean temporal IoU of 0.91 on held-out TTS texts with same-recording references and 0.73 on real-orator recordings with same-recording references.
Observed monotonically decreasing scores from mild to severe flaws in 34 of 35 flaw-and-recording combinations.
Developed speaker-aware normalization to reduce the risk of treating natural vocal differences as delivery problems.
Built an interactive Streamlit dashboard that connects acoustic measurements to understandable feedback.
Improved experiment reproducibility through deterministic processing and dataset integrity checks.

These results demonstrate the strongest performance in controlled, reference-based comparisons while highlighting opportunities to improve generalization.

What we learned

Data quality matters: Accurate temporal labels are essential for meaningful evaluation.
Held-out testing is critical: Calibration performance alone does not reveal how well a system generalizes.
Every voice is different: Speaker-aware normalization is important for separating natural vocal characteristics from possible delivery flaws.
Features need context: A single acoustic measurement can be misleading unless tested against competing explanations.
Reproducibility must be intentional: Deterministic processing, version control, and data-integrity checks make results more trustworthy.
Explanations should be evidence-based: Feedback is more useful when each claim can be traced to an actual measurement.
Limitations matter: Cross-speaker and reference-free analysis remain difficult, and we should communicate those limitations honestly.

What's next for Cadence - Contrastive Speech Delivery Analytics

Our next goal is to make Cadence more robust, accessible, and useful beyond controlled recordings.

Planned improvements include:

Stronger reference-free analysis: Develop better speaker-specific baselines for users who do not have reference recordings.
Better cross-speaker comparison: Improve methods for distinguishing meaningful delivery differences from natural differences between voices.
Broader evaluation: Test more speakers, accents, speaking styles, spontaneous speech, and real-world recording conditions.
More precise localization: Improve word-level alignment and reduce false detections caused by noise or boundary uncertainty.
Actionable coaching: Provide targeted exercises and progress tracking based on recurring delivery patterns.
Greater transparency: Show confidence and uncertainty alongside detected issues so users can interpret feedback appropriately.
Improved accessibility: Build a smoother user experience that helps students, presenters, public speakers, and other learners practice independently.

Our long-term vision is to make speech coaching more precise and transparent, transforming audio measurements into actionable feedback that helps people understand and improve how they communicate.