SpeakIQ AI Coach

Evidence-first speech coach: upload a speech and get every delivery flaw (pauses, pace, pitch, fillers) pinned to exact timestamps, with the measured evidence.

Comment

Inspiration

When I first read the Track C problem statement, it made total sense to me. In speech competitions, people lose marks for delivery, but the feedback is usually something like "work on your pacing" or "too many pauses". Nobody tells you where in the speech it went wrong or by how much.AND yeah my KEEN interest in HIGH TICKET SALES (HTS) drawned me to this topics and yeah one person i am inspired was shelby sapp.

I wanted to build something that doesn't just say "your delivery was weak". It should point to the exact second, show what was measured, and explain why it counts as a flaw. That's how SpeakIQ AI Coach started.

What it does

You upload a speech recording (WAV, MP3, OGG or FLAC). You can also add the transcript and a reference recording of the same text. SpeakIQ then:

  • Measures the audio: energy, spectral features, MFCCs, pitch (F0), pauses and speaking rate
  • Transcribes it locally and gives every word a start and end time
  • Compares only the same words in both recordings, so the comparison is fair
  • Finds delivery issues: rushed or slow delivery, long pauses, missing pauses, uneven pacing, filler sounds and flat pitch
  • Shows each finding with its exact time interval, the measured numbers, the threshold it crossed and its limitations

On the dashboard you can click any finding and it plays that exact part of the audio. There's also an Acoustic Twin page that shows energy and pitch in 3D (with a 2D fallback), an Evidence Coach page with suggestions, and a printable report.

How I built it

The dataset came first. There's no dataset that has a good speech and bad versions of the same speech, so I made my own. I took 3 public-domain LibriVox recordings (3 different speakers) and created 28 flawed versions of each: 7 types of flaws (too many pauses, rushed, slow, uneven pacing, flat pitch, unclear articulation and filler sounds), each at 4 levels from almost perfect to really bad. That gave me 87 samples. Since I added the flaws myself, I know exactly where each flaw is, and the build gives the same result every time.

Backend: Python + FastAPI. I used librosa and Praat (through Parselmouth) for the audio features and faster-whisper for transcription. Everything runs locally, so no paid API keys are needed.

Frontend: React + Vite + TypeScript + Tailwind CSS, with wavesurfer.js for the waveform, Recharts for the graphs and React Three Fiber for the 3D view.

Storage: saving results to Supabase is optional and never stores audio. Each saved result needs its ID and a secret key, and only a hash of the key is stored.

I used Claude Code as my coding assistant throughout the project.

Challenges I ran into

  • Whisper hides pauses. I thought detecting pauses would be easy, but the speech recogniser often stretches a word over the silence, so the pause disappears from the timings. I had to measure pauses from the audio signal itself as well.
  • Filler words go missing. Whisper mostly skips "um" and "uh". With the recogniser alone, only 1 out of 24 fillers was found. Prompting Whisper to keep them changed around 7% of the other words, so I dropped that and wrote an acoustic check instead.
  • Low voices broke pitch tracking. One speaker had a very low voice and the pitch tracker kept jumping octaves. I added a two-pass pitch range that adjusts to the speaker, plus a stability check. If tracking still isn't reliable, the app skips the pitch result instead of showing a wrong one.
  • Different voices shouldn't count as flaws. A naturally high or low voice is not a mistake, so I compare pitch in semitones relative to each speaker's own voice, not in raw Hz.
  • A Windows crash. The transcription engine kept crashing on Windows because of an old C++ runtime. It took me a while to figure out.

Accomplishments that I'm proud of

  • I built my own contrastive dataset of 87 samples with exact flaw timestamps
  • The system caught the injected flaw in 45 out of 54 flawed samples
  • Zero false findings on the clean reference recordings
  • Filler detection went from 1/24 to 16/24 after I added the acoustic check
  • Every finding is tied to an exact time interval with real measured evidence, and nothing is made up
  • It runs fully locally, with no paid API keys

What I learned

The biggest thing I learned is that it's better to say "I can't measure this reliably" than to show a confident wrong number. So every rule in my app either triggers, doesn't trigger, or is withheld with a reason.

I also learned how much signal processing goes into things that sound simple, like finding a pause or a filler, and why being honest about limits matters. My results come from 3 speakers and modified audio, so they show how the rules behave, not real-world accuracy yet.

What's next for SpeakIQ AI Coach

  • Test it on real, natural flawed speech from more speakers
  • Add phoneme-level forced alignment for even more precise timestamps
  • Publish the dataset publicly
  • Deploy the app online so anyone can try it

GitHub: https://github.com/mokshadpoojary9-bot/speakiq-ai-coach Inspiration

Built With

Share this project:

Updates

Submission history