VocalLens

Inspiration

Effective speaking is not just about the words we use. Pace, pauses, pitch variation, energy, and vocal delivery all influence how a message is communicated. Yet, feedback is often subjective: speakers are told to “improve their pace” or “speak more clearly” without knowing exactly where a deviation occurs or what evidence supports it.

VocalLens explores a more measurable and explainable approach to speech-delivery analysis by comparing a reference recording with a participant's delivery of the same text.

What It Does

VocalLens is a contrastive speech analytics prototype that identifies candidate delivery deviations and connects them to measurable acoustic evidence.

Users can:

• Compare reference and participant recordings of the same English text.
• Inspect candidate deviations such as unusual speaking pace, pauses, pitch variation, and energy dynamics.
• Examine temporal intervals and supporting feature measurements.
• Visualize acoustic features and compare scalar metrics.
• Review held-out evaluation results and export analysis results as JSON or CSV.

Instead of presenting only a score, VocalLens aims to explain what changed, where it changed, and which measurable features support the finding.

How We Built It

VocalLens uses React, Vite, and TypeScript for its interactive frontend and FastAPI as the backend interface to an existing Python analysis engine.

The Python pipeline handles audio validation and decoding, alignment where available, acoustic feature extraction, temporal region mapping, deviation detection, evidence reporting, and prototype scoring. The frontend consumes the API results rather than reimplementing the analysis logic in the browser.

The project includes a contrastive dataset with paired recordings, shared transcripts, annotations, and train/validation/test splits. Automated tests, dataset validation, and a reproducible evaluation workflow help verify the implementation.

Challenges We Faced

One of the central challenges was connecting acoustic measurements to temporally grounded explanations without treating every difference from a reference as a mistake.

Reference and participant recordings can have different durations, and their feature plots may use separate time axes. Alignment, temporal boundaries, and feature comparisons therefore need to be handled carefully. The system must also distinguish missing evidence from a genuine zero value and avoid presenting unsupported conclusions as facts.

Another important challenge was evaluating the system honestly. Synthetic fixtures can test whether the pipeline detects controlled changes, but they cannot establish that the same performance will generalize to natural speech.

What We Learned

Building VocalLens highlighted the importance of reproducible data, explicit ground-truth annotations, interpretable acoustic evidence, and transparent evaluation. We also learned that a useful analytics interface must communicate uncertainty and limitations—not just display metrics.

The project reports its current evaluation as a synthetic feasibility pilot rather than proof of real-world accuracy.

What's Next

The current dataset consists of synthetic tonal fixtures and does not yet contain verified real effective-speaker baselines. The held-out evaluation is small, and energy-dynamics detection has a known missed case. The prototype score is not a validated measure of speaking ability.

Future work includes incorporating appropriately licensed natural-speech recordings, validating transcripts and temporal annotations, expanding severity coverage, and evaluating speaker-agnostic detection on a larger and more representative dataset.

Our goal is to develop VocalLens into a more reliable and explainable tool for contrastive speech analysis, where feedback is grounded in evidence and users can inspect the reasoning behind each finding.

Built With

• audio-processing
• data-visualization
• fastapi
• librosa
• numpy
• python
• react
• rest-api
• scipy
• signal-processing
• streamlit
• typescript
• vite