Frostergen

Inspiration

Speaking feedback is often vague: “slow down” or “pause more.” We wanted to make that feedback easier to investigate by showing where two readings of the same text differ and what measurable change occurred.

What we built

Track C Speech Analysis is an offline-first prototype that compares a reference recording with a new take. Its dashboard places the recordings, transcript, word-position chart, and candidate time intervals together. Reviewers can listen to a flagged region and inspect the timing or acoustic evidence behind it. The system supports human review; it does not automatically judge speaking quality.

How we built it

We created a contrastive benchmark from 21 archival speech passages: 21 references and 252 controlled digital edits, for 273 WAV clips in total. Edits introduce a pause, reduce local volume, speed up a region, or slow it down at three programmed strengths. The dataset records transcripts, edit intervals, severity labels, manifests, and speaker-disjoint train, validation, and test splits.

Montreal Forced Aligner provides estimated word boundaries. Our analysis compares word durations, gaps, energy, pitch, and spectral features. An interval detector proposes regions for review, while a separate paired-audio classifier identifies the known edit type and strength in the controlled benchmark. We built a dashboard that makes the measurements and limitations visible.

Results

On the fixed digital-edit test split, the classifier identified the programmed edit type in 104 of 104 clips and strength in 101 of 104. At an interval overlap threshold of 0.5, the detector alone achieved 49.8% F1. The benchmark-only combined pipeline achieved 62.7% F1, with 65.2% precision and 60.4% recall.

Challenges and lessons

The hardest distinction was between detecting a controlled audio edit and evaluating natural human delivery. Digital edits provide exact ground truth, but they can leave processing cues that do not occur in real speech. Automatic word alignments also need listening checks. We therefore show candidate intervals with evidence and state the evaluation scope clearly.

This is a completed hackathon prototype with source code, evaluation, documentation, and a reproducible pipeline. Its reported scores apply to controlled digital edits, not naturally occurring speaking mistakes or human-rated quality. A larger, independently annotated set of consented human recordings is the next step toward validating it for real-world feedback.

Built With

• css
• html
• javascript
• montreal-forced-aligner
• numpy
• openai
• pocketsphinx
• python
• scikit-learn
• scipy
• soundfile