Inspiration
Speaking well is not only about choosing the right words. How we speak—our pace, pauses, pitch, energy, and vocal clarity—can strongly affect how a speech is received. We wanted to build a system that could go beyond simply transcribing speech and actually explain where and why a speaker's delivery could be improved. This inspired us to create VoxTrace AI, a multimodal speech-analysis system designed to identify vocal delivery flaws and connect them to specific moments in an audio recording.
What it does
VoxTrace AI analyzes a speech recording using a combination of audio signal processing, speech analysis, and AI-based techniques. The system extracts features such as:
- Speaking duration and estimated speaking rate
- Pauses and silence patterns
- Vocal energy
- Pitch and pitch variation
- MFCC-based acoustic characteristics
- Speech clarity and delivery patterns Instead of only giving a single score, our goal is to provide actionable feedback and eventually pinpoint the exact timestamp where a problem occurs. Our broader approach is contrastive speech analysis: comparing an ideal delivery with a flawed delivery of the same speech content. This allows the system to learn the difference between effective and ineffective vocal delivery rather than relying only on generic thresholds.
How we built it
The current prototype uses:
Python for the speech-analysis pipeline FastAPI for the backend API Librosa, NumPy and SciPy for audio processing and feature extraction MediaRecorder for browser-based microphone recording HTML, CSS and JavaScript for the frontend The browser can either record speech through the microphone or accept an audio file. The audio is sent to the FastAPI backend, where it is processed and analyzed. We also added support for browser-recorded WebM audio, converting it to WAV before running the acoustic analysis. This made the system work with recordings directly from the browser.
Challenges we ran into
One of our biggest challenges was handling different audio formats. Browser recordings were saved as WebM, while some audio-analysis libraries work more reliably with WAV or other supported formats. We solved this by integrating an FFmpeg-based conversion step into the backend. Another challenge was making the development environment work consistently on Windows, including Python virtual environments, dependencies, and native audio-processing libraries. We also realized that detecting a flaw is only the first step. A useful speech coach should be able to answer: What is wrong? Where did it happen? Why does it matter? And how can I improve it? This led us toward our next stage: forced alignment, temporal flaw grounding, contrastive learning, and timestamp-level explanations.
Accomplishments that we're proud of
We built a working VoxTrace AI prototype that can record or upload speech and automatically analyze important vocal characteristics such as pitch, energy, pauses, speaking rate, and MFCC features. We also implemented a speech-quality scoring system that identifies potential delivery flaws and provides actionable feedback. We are especially proud of successfully connecting the frontend, FastAPI backend, and audio-processing pipeline into one working application, including support for browser-recorded WebM audio. This gives us a strong foundation for adding temporal flaw detection, forced alignment, and contrastive AI analysis next.
What we learned
Building VoxTrace AI taught us that speech analysis is much more complex than simply converting audio into text. We learned how acoustic features can represent different aspects of vocal delivery. For example, pause ratio can be represented as:
Pause Ratio = Total Pause Duration / Total Speech Duration
Similarly, pitch variation and vocal energy can provide useful signals about delivery style. We also learned about the importance of designing a system that gives interpretable feedback rather than treating an AI prediction as a black box.
What's next for VoxTrace AI
Our vision for VoxTrace AI is to evolve from a basic speech scorer into an AI speech coach that can compare ideal and flawed deliveries, identify temporal weaknesses, explain the acoustic evidence behind each finding, and provide personalized recommendations for improvement.
VoxTrace AI — Hear the flaw. Find the moment. Improve your speech.
Log in or sign up for Devpost to join the conversation.