Resonance

Your speech, under a microscope. Resonance hears the exact moment you rush, stall or say "uh", tells you why, and lets you fix it. Your accent is never scored. Your delivery always is.

Comment

Inspiration

"Uhh… hello. So, um, I'm… presenting to you today…"

Three seconds in, you've already decided something about that speaker. We all get judged on delivery, in interviews, debates, classrooms and meetings, yet most of us repeat the same slips for years because nobody ever told us where they happen. Feedback is vague ("be more engaging"), a good coach is rare, and no tool could point to the exact second you lost the room and say why.

We wanted feedback with a timestamp, a cause and a fix. And we didn't want to punish anyone for who they are: your voice is identity; your delivery is a skill.

What it does

Resonance listens to any recording and returns:

  • 🎯 Every slip, pinned to the second: rushed phrases, misplaced or missing pauses, "uh"s, trailing off, sudden loudness, flat pitch, slurred consonants, skipped or misread words
  • 🧠 The cause in plain words: "a 1.4 s pause between 'these' and 'things', inside a phrase where fluent readers don't pause", plus the points it cost and a coaching tip
  • 📈 A score per area (pacing, pausing, intonation, volume, fluency, clarity, text fidelity) and an overall score where one bad habit can't hide behind good ones
  • 🔁 "Try that part again": re-record just that phrase and see before vs after
  • ⚖️ Fairness by design: accent, voice and recording conditions are never scored; a switch turns off fluency scoring for speakers who stammer

How we built it

1. We built the answer key first: the Flawline benchmark. You can't validate a speech judge without known answers, and labelled delivery-flaw data barely exists. So we took studio recordings of real speakers (VCTK, 8 voices from 6 countries) plus President Kennedy's 1962 Rice University speech, and injected 15 delivery flaws across 7 areas, at 5 strengths, exactly where real people make them:

  • placement by grammar (spaCy boundary types), so a pause lands only where no fluent speaker would pause
  • rendering in the speaker's own voice (PSOLA edits, fillers built from their own vowels)
  • quality gates: speech recognition re-checks every removed word, and we listened and rebuilt the edits that sounded fake
  • 6 recording conditions (noise, phone line, echo, MP3…) to stress-test fairness

Result: 563 clips, ~10 hours, every one with an exact time-stamped label. A leakage audit shows a detector that hears only editing marks does barely better than chance (AUC 0.59).

2. The engine listens audio-first. Every recording becomes measurements 100 times a second (pitch in semitones from the speaker's own median, loudness from their own level, consonant energy, rate, silence), all speaker-normalised. Speech recognisers smooth over hesitations, so we find pauses and fillers in the sound itself and use the words only to explain them. With a clean reading of the same text it aligns the two (DTW, condition-matched); without one it uses the speaker's own baseline and clean-speaker norms.

3. Scoring. Each flagged moment costs $$p = w \cdot c \cdot r \cdot s^{2} \cdot m$$ (flaw weight, recording confidence, detector reliability, severity squared, duration). Each area scores \(100\,e^{-P/35}\), and the overall is \(0.6 \times \text{mean} + 0.4 \times \text{weakest area}\).

4. The app. FastAPI and a hand-built web front end: upload, timeline with feature lanes against the expected range, numbered flaw regions, flaw cards with playback, retake comparison and a downloadable report.

Results (on the Flawline benchmark, with a clean reading to compare against)

  • Flags land a typical 18 ms from the true start
  • 98% name the right area
  • The score falls steadily as a flaw gets worse for 12 of 13 flaw types
  • Finds about 2 in 3 hidden flaws on a speaker it never trained on
  • Fewer than one false flag every two minutes on clean speech in noisy, phone-line or echoey recordings

Everything is reproducible: make setup && make eval && make app, a Dockerfile, a pinned lock file, and checksums for every clip.

Challenges we ran into

  • Making flaws sound human. Our first rushed phrases sounded like network lag and our first "uh"s like random audio. Listening to every pair and rebuilding the renderers was the biggest time sink.
  • Speech recognisers hide the very slips we hunt. Whisper stretched words across pauses and silently dropped "uh"s. Testing on real recordings exposed it, and we rebuilt upload mode to trust the audio, not the transcript, for timing.
  • Not cheating ourselves. We split by speaker, fitted only on train, kept the test speakers untouched, and audited whether editing traces leak the answer.
  • Fairness. Noise, echo or an accent must not look like a flaw, so we built invariance tests and measure them.

What we learned

A benchmark is only as good as its realism, and a detector is only as good as the benchmark that checks it. Building both together, the contrastive way, made every claim measurable. We also learned to report honestly: upload mode with no reference is harder, and we publish those numbers too.

What's next

  • A gold set of real speeches labelled by debate coaches
  • Indian English and code-mixed speech, so it works for the way millions of us actually talk
  • Coaching written by a language model, grounded in the measured evidence

Built With

Share this project:

Updates

Submission history