Inspiration
Track C asked for contrastive speech analytics: compare a speaker against a reference and ground every flaw in time. Most speech feedback tools give one overall score. I wanted each problem tied to specific words and seconds, with the measured numbers behind it. I figured it would be something interesting to do as a challenge.
What I learned
Comparing two different people is much harder than comparing a recording with an edited copy of itself. Two good readers naturally differ a lot, so the thresholds have to be wide or everything gets flagged. I also learned to fix the metric before looking at test results, and to back a cause with a test: undoing the edit should remove the flag, and a fake edit shouldn't change anything.
How I built it
Both recordings are aligned word by word with a wav2vec 2.0 aligner. Then nine things are measured on the same words in both: pace, pauses, pitch range, emphasis, volume, clarity, word time, sentence endings and speech tempo. Pitch and volume are relative to each speaker's own voice, so a deep voice or a quiet mic isn't counted as a mistake.
There's no trained classifier. Each measure has a threshold, calibrated on comparisons where nothing is wrong:
$$z = \frac{x}{s_k}, \quad \text{flag if } z \ge h$$
To test it I built a dataset of 2,410 audio files: 9 public-domain speeches, 448 copies with deliberate edits (rushing, dropped pauses, monotone, fading and more, each with known words and timing), and 1,953 readings by other people. On speakers it never saw, it gets an F1 of 0.812. Undoing an edit removed its flag in 22 of 22 tests.
"Hear it fixed" edits your own recording toward the reference and runs the analysis again. The app is FastAPI and plain JavaScript, it runs in Docker, and the detector is on Hugging Face.
Challenges
- Different readers produce false flags, about one every two minutes, and more for non-native speakers. I couldn't get rid of that, so I report it.
- The aligner sometimes shifts word boundaries when only the volume changes, which can look like a timing problem.
- The first version of "Hear it fixed" processed the whole clip and created new flags by itself, so it now edits only the flagged part.
- Making a flat voice more expressive with your own pitch barely worked, so there's an option to borrow the reference's melody instead.
- People can paste the wrong transcript, so the app checks the text against the audio and says where it doesn't match.