SpeechLens — Hear Beyond the Words
Inspiration
As an engineering student, preparing for internships or applying for first job as freshers it involves more than just having technical concepts. We build projects, basic fundamentals everything, even practice interview questions, and prepare our answers. Yet, many students when they face interviewer, communicating is the key , we should be able to deliver with confidence and clarity so it's the challenging part.
Many students are technically capable but struggle with rushing, hesitation, unclear speech, or flat delivery while speaking if they are under pressure. The difficult part is for them they may know it could have been better but they might not know exact reason behind it like was it because the answer was rushed? was it unclear? was there unnecessary pauses? and lot more.
Feedback such as "be more confident" or "improve your communication skills" often does not explain properly what should the student change in their practice because their respective speech is not analysed properly. So, to improve this gap it inspired us to build SpeechLens.
Our goal is not just just to simply assign scores, but to help speakers understand where a potential issue might occur, what evidence it supports, so it looks beyond words being spoken and examines how it was delivered. It has many forward potential applications like in presentations, public speaking, classroom, Ted-Talks, and other situations where effective verbal communication does matter.
What it does
A score tells you how you performed. SpeechLens helps you discover the moment that made the difference. It transforms speech into a time-aware, evidence-driven analysis of delivery. Instead of treating an entire recording as one score, it compares delivery characteristics against a reference/baseline, according to our dataset, examines speech over time, and identifies potential issues to connect those moments in the recording. It helps user investigate. -Hesitation and pauses: Interruptions in speech if they occur within phrase or are they natural. -Rushing and dragging: Identify changes in speaking pace that may affect clarity. -Monotone Delivery: limited vocal expressiveness. -Mumbling and vocal clarity: Investigate acoustic patterns that may indicate reduced clarity. -Temporal delivery patterns: Explore when a potential issue begins.
What makes SpeechLens different is the connection between detection and understanding. A pause is not automatically a flaw. A change in pitch is not automatically a problem. SpeechLens is designed to help users investigate the context instead of treating every deviation as a mistake.
How we built it
SpeechLens has been designed as an end-to-end machine learning pipeline that translates raw speech input into measurable acoustic evidence, predictions from models, timestamps for delivery problems, and valuable coaching feedback. Instead of basing our work on a single number or black-box prediction, we have employed acoustic feature engineering, word-level temporal alignment, comparative modeling, and evidence analysis.
The design of SpeechLens comprises two inter-related processing pipelines — audio processing and transcript alignment. They come together in temporal analysis and model inference.
Audio Preprocessing and Standardization
We began by preparing the audio recordings for consistent analysis. Preprocessing establishes a common representation of the input signal so that recordings can be analyzed and compared more reliably.
For a waveform containing (N) samples at sampling rate (f_s), its duration is
[ T=\frac{N}{f_s} ]
We also accounted for differences in recording length and signal characteristics when preparing the inputs for feature extraction. The exact normalization and scaling operations were kept consistent with the implemented pipeline.
Audio Pipeline: Acoustic Feature Extraction
The audio pipeline derives measurable properties of speech, which can be used to separate good delivery from bad.
We employ signal processing and acoustic analysis techniques to describe speech using numerical parameters instead of viewing an audio file as an opaque input. The type of feature set employed will determine what can be derived through measurements from the audio file. For example, root mean square (RMS) energy measures the signal's average magnitude:
[ \operatorname{RMS}(x)= \sqrt{\frac{1}{N}\sum_{n=1}^{N}x[n]^2} ] It is important to look for any patterns that relate to delivery but should understand that a single acoustic measurement alone will not be able to prove a speech error.
Transcript Pipeline: Forced Alignment and Temporal Grounding
The transcript pipeline links the speech data with the timestamp. We leverage the word-level aligner information to link the transcripts to their approximate timestamp. An aligned transcript can be represented as
[ A={(w_i,s_i,e_i)}_{i=1}^{N_w} ]
where (w_i) is a word and (s_i) and (e_i) are its estimated start and end times.
The duration of an aligned word is
[ d_i=e_i-s_i ]
Dataset Preparation and Contrastive Learning Setup
We created a dataset containing effective speech and purposefully incorrect speech in order to test and train our supervised machine learning classifiers. We tested performance on two separate but complimentary levels, one being "recording level classification "to test the overall quality of delivery and "frame-level classification" for localization.
Model Training and Comparative Evaluation
In our work, we experimented with three types of models for speech analysis at both recording level and frame level: Random Forest, Support Vector Machine (SVM), and HistGradientBoosting. We chose HistGradientBoosting to be used in further experiments due to its high accuracy compared to other models.
The comparative approach helped us to choose the model according to the results.
Connecting Predictions to Timestamped Speech Events
Potential areas of poor delivery are determined by leveraging the acoustic characteristics, model output, timestamp per word, and transcribed context in combination.
Duration of an event starting at t_s and ending at t_e is given as:
[ \boxed{D=t_e-t_s} ] Where applicable, we compare delivery measurements against a reference baseline to identify meaningful deviations. Model output at the recording level provides a summary measure of the delivery quality of the speaker, whereas at the frame level, it helps identify localized patterns. By anchoring model output to time and acoustic evidence, SpeechLens goes beyond identifying a pattern to showing where it exists and why it could be important.
Explainable Feedback and Interactive Dashboard
The dashboard provides actionable feedback by transforming predictions from models and acoustic measurements. It combines information on overall performance, time-stamped transcript, temporal understanding, and evidence that could be related to possible problems with delivery.
SpeechLens wants to provide answers to three main questions for each problem spot: What is the problem? Where does it occur? How can I improve? The connection of the feedback with certain times and evidence will help users go beyond general recommendations.
Testing, Validation, and Limitations
We used a set of passing tests to validate the behaviors of the pipeline, one of which is a synthetic pause timestamp test. The alignment scores and unusual timing patterns are also signals when it comes to examining potential trouble areas.
This is a measure to ensure pipeline reliability but not necessarily a surefire way of detecting all problems in the speech. The SpeechLens can differentiate between acoustic measurement versus interpretation when it comes to speech delivery.
Challenges we ran into
One such difficulty was that we had to use a small number of labeled data, which made it difficult to train models that generalized well to unseen data.
The other difficulty faced was that a single acoustic pattern can have multiple interpretations. For instance, a three-second pause may be a sign of hesitation while speaking in an interview, or it could be just breathing. Likewise, a high speaking rate could be because of nervousness or simply the person’s own way of speaking.
This presented us with the following problem: not all variations in speech delivery are necessarily problems. One person may make purposeful pauses, and have a slower speech rate; another may have a quick rate with emphatic delivery. However, both could deliver their message successfully. Therefore, SpeechLens had to differentiate between possible problems and normal variations in speech delivery styles rather than considering one delivery style the standard.
Another problem we had to solve was how to connect the description of the entire speech segment and pinpoint the exact time where it should be fixed. The person may hurry with the one sentence only, while the rest of the sentences will be delivered well.
In the end, the task was not just to point out anomalies in the way someone speaks, but to understand the background behind it, respect their unique style of speaking, and help them improve in the right areas.
Accomplishments that we're proud of
Among our greatest accomplishments was the development of the full pipeline of SpeechLens, ranging from data preparation and training of the models all the way up to acoustical analysis and temporal grounding. The training and integration of the models in tandem with the audio processing and alignment components meant facing and overcoming several technical challenges.
Our greatest pride is the establishment of the groundwork for speech feedback beyond a single aggregate score, which makes a link between model predictions and actual points in time within the recording.
What we learned
Working on SpeechLens made us realize that predictions are valuable insofar as the evidence that supports them is comprehensible by users. It was one thing to identify an issue in a speech recording but another to display it in a meaningful way. We learned how to tie each piece of information to its timestamp, corresponding text excerpt from the transcript, and the evidence of acoustic or timing nature that led to detection of the issue. Additionally, we have realized how important the process of interpretation of technical results is for successful communication with a user. The result of a certain model should not be a label but a piece of evidence that allows one to investigate the situation.
What's next for SpeechLens — Hear Beyond the Words
-Individualized Feedback: Compare individual practice sessions against each other in order to monitor the changes in the speed of speech, pauses, delivery style, etc., while pointing out recurring problem areas.
-Real-Time Training Mode: Provide soft reminders in case of an excessive pause, hurried speech, or constant hesitation in order to help users correct their delivery during training.
Multisensor Analysis: Analyze speech alongside other signals such as eye contact and body posture in order to get a fuller picture of the delivery in interviews or presentations.
-Language Support: Move past English and provide language-specific analysis in terms of transcription, synchronization, and evaluation to ensure accuracy in different accents and speech patterns.
-Individual Vocal Signature and Drift: Identify the signature of each speaker and point out any significant deviations from their usual speech pattern in terms of speed, intonation, etc.
-Individual Filler-Words Fingerprint: Measure the usage of filler words like “um,” “uh,” and “like” by providing their frequency and timing data.
SpeechLens turns speech analysis into actionable, personalized insights, helping every speaker understand their delivery and communicate with greater clarity and confidence.
Log in or sign up for Devpost to join the conversation.