VaaniSutra

Inspiration

Project Story : We have all been there attending many speech events and noticed the same flaws again and again, filler words, wrong pace, and long, awkward pauses. In our own class presentations, we stumbled too. We knew what was wrong, and we promised to improve. But every time, we failed. The problem was clear, we knew we made mistakes, but we never knew exactly where, or how to fix them. So we built [name], a web app where you practice before you present. It compares your speech with an ideal one, finds flaws at exact moments, gives a score, and tells you how to fix them. Our belief, every mistake should make the next speech better.

What it does

VaaniSutra is an AI speech coach. You speak or upload audio, and it: Gives you a score out of 100 Marks every flaw (long pauses, fillers, fast pace) with its exact time Shows you how to fix each one Tracks your progress over time It compares your speech with ideal and flawed examples, so the score is fair and the same every time.

How we built it

Collect the dataset Record about 20 ideal and 20 flawed speeches. Use the same script for both, so the comparison is fair.
Convert speech to text Use Whisper (faster-whisper). It gives every word with its start time, end time, and confidence.
Measure the speech From that data, calculate: Pace: words per minute Pauses: gaps between words Fillers: um, uh, matlab Clarity: how sure Whisper was about each word Confidence: pitch and energy (using librosa)
Score it (contrastive) Compare your numbers with the ideal and flawed averages from the dataset. Close to ideal means a high score. Close to flawed means a low score. No randomness, so the score is always the same.
Find flaws with timestamps Simple rules, for example: gap over 1.2 s means a long pause. Each flaw is saved with its start and end time.
Generate tips Each flaw type has a ready tip, like "Take a breath before your next point."
Build the dashboard React app with a score gauge, a waveform with flaw markers, a progress chart, and an ideal-vs-flawed comparison. Tech stack Backend: Python + FastAPI AI/Audio: Whisper, librosa Database: SQLite Frontend: React + Tailwind + Recharts Flow: Audio, then Whisper, then metrics, then score, then flaws and tips, then dashboard.

Challenges we ran into

No ready dataset: no public "ideal vs. flawed" speech data, so we built our own. Fair, repeatable score: we used a fixed formula, not an LLM, so the same audio always gets the same score. Exact flaw timing: we used Whisper's word timestamps to mark each flaw at the right second. Windows setup issues: npm and audio library errors took time to fix. English + Hindi mix: we added fillers like "matlab" and "toh" to catch real speech. Slow analysis on laptops: we used a smaller Whisper model to speed it up.

Accomplishments that we're proud of

Built a complete working app in a short time: record, analyze, score, and dashboard, all end to end. Created our own contrastive dataset of ideal vs. flawed speeches, which no public dataset offers. Reproducible scoring: the same audio always gives the same score, backed by automated tests. Flaws pinned to the exact second, so users see where they went wrong, not just a number. Clear ideal vs. flawed gap: our validation shows ideal speeches scoring much higher than flawed ones. Works with English and Hinglish, like real speakers talk. Solved a problem we faced ourselves, and turned our own stumbles into a tool that helps others.

What we learned

You can't fix what you can't measure. Real feedback needs numbers and timestamps, not just "it was good." Data matters more than code. A good ideal vs. flawed dataset made our scoring meaningful. Reproducibility builds trust. A fixed formula beats a score that changes every time. Speech has many parts. Pace, pauses, fillers, and clarity each need their own measure. Real-world setup is hard. Windows errors, audio tools, and slow models taught us to debug patiently. Start simple, then improve. Synthetic data got us running, and real recordings made it better. Our own mistakes were our best teacher. Building this made us better speakers too.

What's next for VaaniSutra

Bigger real dataset: record more speakers, accents, and languages (Hindi, Marathi, English, Hinglish). Live feedback: real-time tips while you speak, not just after. More Indian languages: full support beyond English and Hinglish. Body language analysis: use the camera to check eye contact, posture, and gestures. Personal coach plans: custom practice exercises based on your weakest areas. Interview and presentation modes: practice with real questions and topics. Mobile app: practice anywhere, on your phone. Partnerships: work with colleges and training centers to help students speak with confidence.