Inspiration
Speaking well is more than saying the right words. How we speak—the pauses we take, the pace we maintain, and the energy and variation in our voices—can fundamentally change how a message is communicated.
Yet feedback on public speaking is often subjective. A speaker might be told to "improve their pacing" or "sound more expressive" without being shown exactly where the problem occurred or what measurable evidence supports that feedback.
We wanted to approach this problem differently.
Instead of building another AI speech coach that produces a score and generic recommendations, we asked a more fundamental question:
Can we build a system that knows exactly where a speaking flaw was introduced, independently detects it, and measures how accurately it identifies the mistake?
This question inspired FlawBench, a ground-truth-driven approach to speech analytics that combines controlled audio transformations, acoustic signal processing, temporal flaw detection, and quantitative evaluation.
Our goal is to make speech evaluation more transparent, measurable, and reproducible.
What it does
FlawBench is a speech-analysis system built around Parametric Flaw Injection with Exact Ground Truth.
It starts with a clean reference recording and generates controlled variations containing deliberately introduced delivery flaws. Because the transformations are generated programmatically, the system can record the exact parameters and affected time intervals.
FlawBench focuses on three initial categories of acoustic deviations:
- Excessive pauses: Introducing silence at controlled locations to examine whether the system can identify abnormal pauses.
- Pacing deviations: Changing the speed of a selected speech segment to test whether local timing differences can be detected.
- Energy reductions: Reducing the signal amplitude over a selected interval to examine whether localized energy changes can be identified.
The system then analyzes the reference and modified recordings independently of their ground-truth annotations.
Its workflow connects several stages:
- Controlled dataset generation: Create modified recordings with documented transformation parameters and known temporal boundaries.
- Acoustic feature extraction: Analyze characteristics such as RMS energy, energy in decibels, spectral features, MFCCs, pitch, and speech activity.
- Reference comparison: Compare the original and modified recordings to identify meaningful temporal and acoustic differences.
- Flaw detection: Identify candidate intervals corresponding to pause, tempo, and energy deviations.
- Ground-truth evaluation: Compare predicted intervals with the known annotations using metrics such as precision, recall, F1-score, and temporal Intersection over Union (IoU).
- Evidence-based analysis: Connect detections to the underlying measurements so that results can be inspected rather than accepted blindly.
The central idea is that every generated flaw has a traceable origin, every prediction can be evaluated against an independently stored annotation, and every reported metric should be reproducible from actual data.
How we built it
We organized FlawBench as a modular Python application so that dataset generation, signal processing, detection, evaluation, and visualization could evolve independently.
1. Dataset engineering and ground truth
We developed a dataset-generation workflow that preserves reference recordings and creates controlled variants using configurable transformation parameters.
Each generated sample is associated with metadata describing its source, transformation type, parameters, and affected interval. Checksums and provenance information help identify the actual audio files used during processing.
A consistent timestamp convention is essential. The system records intervals using sample indices and maps them to the modified recording's timeline, accounting for the duration changes introduced by pauses and tempo transformations.
2. Acoustic feature extraction
We built the feature-extraction layer around established digital signal-processing techniques.
The pipeline extracts time-indexed acoustic measurements and preserves the information needed to compare recordings. It accounts for sampling rates, frame boundaries, silent regions, and undefined pitch estimates.
Rather than normalizing away every difference, the pipeline preserves measurements that may reveal the injected flaw.
3. Independent flaw detection
The detection layer analyzes the recordings and their acoustic comparisons to identify candidate flaws.
The pause detector examines silence and duration differences. The tempo detector investigates sustained differences in temporal progression. The energy detector compares corresponding energy measurements.
Each prediction records its flaw category, interval, and supporting evidence.
Crucially, the detector does not receive the known ground-truth annotations as input. Those annotations are reserved for evaluation.
4. Quantitative evaluation
We designed the evaluation layer to compare independently generated predictions with the dataset's known annotations.
It evaluates event matching, temporal overlap, classification performance, and localization errors. It also distinguishes genuine zero-detection results from missing or failed analyses.
This lets us investigate not only whether a flaw was detected, but also how precisely its location was identified.
5. Application and tooling
We used Python for the core processing pipeline, numerical audio libraries for signal analysis, pytest for automated testing, and Streamlit for the interactive interface.
The architecture includes centralized configuration, reusable modules, command-line workflows, data validation, and reproducibility checks.
The implementation is divided into phases so that each layer can be tested before the next one depends on it.
Challenges we ran into
Getting timestamps right
A pause insertion changes the timing of everything that follows it. A tempo transformation can change the duration of the modified segment and shift subsequent events.
Simply copying the original timestamps into the modified recording would therefore produce incorrect ground truth.
We addressed this by treating sample indices as the authoritative representation of time and recording the affected intervals in the output recording's timeline.
Preventing ground-truth leakage
Because FlawBench knows which transformations it applied, it would be easy to accidentally build a detector that simply reads the expected answer.
That would defeat the project's purpose.
We therefore separate generation, detection, and evaluation. The detection layer receives audio and permitted acoustic evidence, while the known annotations remain reserved for scoring.
Distinguishing genuine acoustic differences from meaningful flaws
Not every difference between two recordings represents poor delivery. Speakers naturally differ in pitch, energy, and timing, and even the same speaker may perform a sentence differently.
We learned to distinguish measurable acoustic changes from interpretations of those changes. Detection results must be tied to explicit comparison rules and should not automatically be treated as proof of poor speaking quality.
Making evaluation scientifically meaningful
A detector can appear successful if training and testing use nearly identical recordings. Multiple variants of the same source can also create misleading results when distributed across different data splits.
Our evaluation design therefore emphasizes source-level separation, explicit event-matching rules, and transparent reporting of false positives, missed events, and localization errors.
Validating beyond controlled transformations
Synthetic transformations provide exact ground truth, but they do not capture every characteristic of naturally occurring speech.
A detector that recognizes an artificially inserted silence or energy reduction may not generalize equally well to spontaneous speech.
Establishing that distinction is important to interpreting our results honestly.
Accomplishments that we're proud of
We are proud of building FlawBench around verifiable evidence rather than treating an AI-generated score as the final answer.
Our principal accomplishments include:
- Designing a controlled contrastive dataset-generation approach with known flaw locations.
- Establishing a structured ground-truth format with explicit temporal boundaries and transformation provenance.
- Building modular components for acoustic feature extraction and reference-versus-variant comparison.
- Developing independent detection workflows for pause, tempo, and energy-related deviations.
- Designing an evaluation pipeline that compares detections with annotations using established classification and temporal-localization metrics.
- Incorporating validation, configuration management, automated testing, and reproducibility into the development process.
- Building the foundation for an interactive interface that connects detected intervals with their acoustic evidence.
Most importantly, we have a methodology for testing whether the system identifies the mistakes it was designed to detect, instead of relying only on qualitative demonstrations.
What we learned
We learned that building a trustworthy AI system requires more than selecting a model and displaying its predictions.
Ground truth is an engineering problem. Accurate labels depend on careful handling of audio transformations, sample boundaries, timestamps, and data provenance.
Evaluation must be independent. A detection result is meaningful only when the system has not been given the answer it is supposed to predict.
Explanations need evidence. An observed change in pitch, energy, or timing is a measurable fact. Interpreting that change as a speaking flaw requires additional context and an explicit rubric.
Reproducibility should be designed from the beginning. Configuration, transformation parameters, checksums, and version information make it easier to investigate differences between runs.
Limitations are part of trustworthy AI. Controlled synthetic tests are valuable, but they cannot establish real-world performance on their own. We need broader speech recordings, more varied speakers, and eventually human evaluation to understand how well the system generalizes.
These lessons shaped FlawBench into an evidence-oriented research prototype rather than a generic speech-scoring application.
What's next for FlawBench: Ground-Truth Speech Analytics
Our next goal is to turn the technical pipeline into a more complete participant-facing speech-analysis experience.
We plan to strengthen the connection between detection and understandable feedback by adding:
- Synchronized reference-versus-participant audio playback.
- Interactive timelines that highlight detected flaw intervals.
- Feature visualizations that let users inspect the acoustic differences behind each result.
- Evidence-linked explanation cards that connect a detection to actual measurements.
- Contextual recommendations that help users review specific sections of their delivery.
- Exportable analysis reports containing the actual predictions, evidence, and limitations.
We also want to expand testing beyond controlled audio transformations by incorporating appropriately licensed or self-recorded speech with carefully reviewed annotations.
Future work will investigate speaker-agnostic normalization, stronger temporal alignment, additional delivery-flaw categories, and human-reviewed evaluation.
Our long-term vision is a system where speech feedback is not simply a judgment such as "your pacing needs work," but an inspectable explanation of what changed, where it changed, why the system flagged it, and how accurately the system can identify similar problems.
FlawBench aims to make speech analytics measurable, reproducible, and explainable—one ground-truth flaw at a time.
Log in or sign up for Devpost to join the conversation.