SpeechLens

• 

  audio result

• 

  DashBoard Score

• 

  pitch-energy

• 

  transcript

• 

  Upload and analysis settings

• 

  waveforms

Inspiration

Public speaking is not only about what a person says, but also how they say it. Pace, pitch, pauses, volume, and vocal clarity can significantly change the quality of a delivery.

For this challenge, we wanted to build a system that could go beyond a simple speech score and answer a more useful question:

Where exactly did the speaker's delivery go wrong, what changed compared with a good reference, and why was it considered a flaw?

This led us to build SpeechLens, a contrastive speech-delivery evaluation system for Multimodal AI Hackathon 2026 – Track C.

What We Built

SpeechLens compares a participant's speech against a reference delivery of the same transcript.

The system:

1. Accepts reference and participant audio along with the transcript.
2. Performs speech-to-text and forced alignment to obtain word-level timestamps.
3. Extracts acoustic features such as:
   • Pitch / F0
   • Energy / volume
   • MFCCs
   • Spectral features
   • Speech rate
   • Pause patterns
   • Vocal clarity
4. Normalizes speaker-dependent characteristics so that the comparison focuses on delivery differences.
5. Temporally aligns the reference and participant features.
6. Detects delivery flaws such as:
   • Excessive pauses
   • Monotone delivery
   • Volume drops
   • Unusually fast or slow speech
   • Reduced vocal clarity / mumbling
7. Grounds detected flaws to specific time regions.
8. Converts the measured acoustic differences into human-readable explanations.
9. Produces a reproducible overall delivery score.
10. Visualizes the complete analysis through an interactive Streamlit dashboard.

The core idea is to connect a measurable signal change → temporal region → detected flaw → human-readable explanation.

How We Built It

SpeechLens was implemented primarily in Python.

For audio processing, we used libraries including Librosa, NumPy, SciPy, SoundFile, Praat-Parselmouth, and PyWorld. WhisperX is used for speech recognition and alignment, while Plotly and Streamlit power the interactive analysis dashboard.

The project was developed in multiple stages.

First, we created controlled synthetic speech transformations to generate different levels of delivery flaws. This allowed us to test the detection pipeline without depending entirely on manually collected flawed recordings.

Next, we implemented feature extraction and speaker-aware normalization. We then built the comparison and detection pipeline using temporal alignment and configurable thresholds.

Finally, we integrated everything into an interactive dashboard where users can upload audio, configure the analysis, inspect waveforms and acoustic features, navigate detected flaw regions, view transcript-level highlighting, and export results.

Challenges

One of the biggest challenges was temporal grounding. Detecting that a speaker has a problem is not enough; the system needs to identify when that problem occurs.

Another challenge was separating actual delivery differences from natural differences between speakers. Pitch and energy can vary substantially between people, so speaker-aware normalization was necessary.

We also had to make the detection system reproducible rather than relying on subjective judgments. This required configurable thresholds, persistence rules, temporal merging, and explicit scoring weights.

During deployment, we also encountered dependency compatibility issues involving pyworld and setuptools. We resolved the deployment dependency conflict and successfully deployed the dashboard on Streamlit Cloud.

What We Learned

Building SpeechLens helped us understand how digital signal processing, speech alignment, feature engineering, and visualization can work together in a real application.

We learned that a useful AI system should not only produce a prediction or score, but should also provide evidence for that result.

The project also taught us the importance of reproducibility, controlled test data, temporal alignment, speaker normalization, and designing an interface that makes technical analysis understandable to users.

AI-Assisted Development

GitHub Copilot was used during development to assist with implementation, debugging, documentation, and testing. The project logic, architecture, validation, and final integration were reviewed and tested by the team.

Result

SpeechLens turns speech evaluation into an interactive and explainable analysis rather than a single black-box score.

Instead of only saying "your speech needs improvement," SpeechLens tries to show:

What changed → Where it changed → Which delivery flaw it represents → Why the system detected it.

Built With

• copilot
• librosa
• numpy
• plotly
• praat
• python
• scipy
• streamlit
• whisperx