🎙️ SpeechProof — Every Word. Every Pause. Fully Auditable.
Auditable speech scoring with verifiable acoustic evidence, an immersive audio-reactive interface, and an adversarial Crash Test Lab.
💡 Inspiration
Most speech evaluation tools provide a score without explaining exactly why a speaker received it. A user might receive a delivery score of 74/100, but the feedback may not identify the specific pauses, speech-rate changes, or vocal-energy variations behind that result.
We wanted to explore a different approach: What if speech scoring could be inspected, reproduced, and stress-tested rather than simply trusted?
That question inspired SpeechProof, a research platform that connects measurable acoustic features to timestamped evidence and lets users challenge the scoring rubric through controlled audio transformations.
Our goal is to make speech evaluation more transparent, reproducible, and scientifically honest.
🚀 What SpeechProof Does
SpeechProof combines speech analysis, evidence visualization, and adversarial testing in one application.
- Universal Audio Ingestion: Upload WAV, MP3, M4A, AAC, FLAC, OGG, WebM, MP4, and Opus recordings. Supported inputs are normalized to canonical 16 kHz mono PCM WAV.
- Live Microphone Recording: Record speech directly in the browser, with an elapsed timer, live audio-level monitoring, and automatic audio normalization.
- Speech Recognition: Use faster-whisper with the large-v3 model to generate transcripts and word-level timestamps.
- Acoustic Scoring: Calculate measurable delivery features, including speech rate, pausing, pitch variation, and vocal-energy variability.
- Interactive Evidence Inspector: Explore waveform regions, inspect timestamps and measurements, view surrounding word context, and seek directly to the corresponding audio.
- Contrastive Comparison: Compare reference and candidate recordings to inspect changes in measured delivery features.
- Crash Test Arena: Inject controlled pauses, modify speech pace, and alter volume to investigate whether the scoring rubric responds as expected.
- Tamper-Evident Run Ledger: Store evaluation records in SQLite and calculate SHA-256 hashes for audio and evidence integrity.
- Validation Matrix: Examine the status of research evaluations, including deterministic reproducibility, dose-response behavior, cross-talk, and human agreement.
🧠 How We Built It
SpeechProof uses a React 19, TypeScript, and Vite frontend connected to a Python FastAPI backend.
The interface features a custom, Siri-inspired Living Audio Orb implemented with a high-DPI HTML5 Canvas. During microphone recording, the visualization responds to actual audio input. Its appearance changes between idle, recording, analyzing, and results states.
The backend uses faster-whisper, CTranslate2, and CUDA float16 for local speech recognition when the required model and compatible GPU environment are available. Audio processing uses libraries such as PyAV, Librosa, SoundFile, NumPy, and SciPy.
Uploaded recordings pass through an audio-ingestion pipeline that decodes supported containers, downmixes channels when needed, resamples audio, and validates the canonical representation before scoring.
The scoring engine produces acoustic measurements and timestamped evidence. FastAPI endpoints expose analysis, comparison, audio transformation, run history, and validation functionality.
We use automated tests to verify software behavior and SQLite with SHA-256 hashes to support reproducibility and evidence-integrity checks.
🛠️ Challenges We Faced
1. Handling different audio formats
Recordings can use different codecs, sample rates, channel counts, and containers. We built a multi-stage decoding and normalization pipeline to make supported recordings usable without requiring users to convert everything manually.
2. Connecting visual feedback to real audio
The Living Audio Orb needed to respond to microphone input without making the interface sluggish. We integrated browser audio analysis with an animation loop and added distinct visual states for recording and analysis.
3. Making evidence inspectable
A numerical score is difficult to evaluate without context. We connected waveform regions to timestamped measurements and surrounding transcript information, allowing users to investigate what the system actually detected.
4. Avoiding misleading fluency scores
We identified a calibration limitation in our fluency dimension. Rather than treating an unreliable measurement as a valid zero, SpeechProof explicitly marks the dimension unavailable and avoids imposing a false-zero penalty on the overall score.
5. Separating software correctness from research validity
Passing automated tests does not establish that a scoring rubric is valid for a broad population. We therefore distinguish software verification from empirical findings and disclose where additional participants, repeated recordings, and human ratings are required.
📚 What We Learned
Building SpeechProof taught us that reliable audio analysis involves much more than running a speech-recognition model.
Audio normalization, temporal alignment, metric definitions, reproducibility, error handling, and user experience all influence the final result.
We also learned the importance of testing algorithms against controlled perturbations instead of assuming their outputs are correct. A useful scoring system should not only produce results; it should make those results inspectable and expose uncertainty and limitations.
Finally, visual polish matters most when it supports real functionality. Our Living Audio Orb connects the recording experience to the underlying audio rather than serving only as decoration.
🔬 Research Results and Limitations
The current backend test suite has 113 reported passing tests, with a successful reported frontend production build.
Our research evaluation is more limited than our software verification.
The current empirical pilot is based on a single authorized recording. Some evaluation properties have pilot findings, while others remain inconclusive or unrun because the required repeated takes, multiple speakers, or human-rating datasets are not yet available.
We do not claim that every empirical evaluation has passed or that our scores are validated for general populations.
These limitations are part of the research, not results we want to hide. SpeechProof is a research prototype intended to make speech scoring auditable, not a system for high-stakes hiring, admissions, or clinical decisions.
🔐 Privacy and Responsible Use
When run locally, speech recognition and acoustic feature processing execute on the user's machine rather than relying on third-party cloud speech APIs by default.
Recordings should only be processed with appropriate speaker consent. Evaluation results should be interpreted in context, and the current rubric should not be treated as a definitive judgment of a person's communication ability.
🤖 AI-Assisted Development
We used Google Antigravity as an AI coding assistant during development and ChatGPT for development assistance, planning, and documentation.
These tools supported implementation and refinement, but the team remains responsible for understanding, testing, and explaining the submitted code.
The speech-recognition engine, faster-whisper, is a component used by the application; it is distinct from the AI coding assistants used during development.
🔮 Future Work
Future improvements include collecting a larger multi-speaker evaluation dataset, obtaining independent human ratings, investigating empirical calibration, expanding cross-hardware reproducibility testing, and evaluating the rubric across a broader range of speaking styles and recording conditions.
SpeechProof — Every Word. Every Pause. Fully Auditable.
Log in or sign up for Devpost to join the conversation.