VoxTrace: Evidence-Grounded Speech Analytics

Turn speeches into measurable feedback with reproducible rubric scores, audio-timestamped findings, and ideal-versus-flawed comparisons—all with a zero-cost local analysis mode.

Comment

VoxTrace: Every Score Has Evidence

Inspiration

Public-speaking feedback often sounds vague: “Speak more clearly,” “Improve your pacing,” or “Make your argument stronger.” But useful feedback should explain what needs improvement, why it matters, and where the issue occurred.

For Track C of the Multimodal AI Hackathon 2026, we built VoxTrace, a speech evaluation system designed to make feedback more transparent, reproducible, and actionable.

Our central idea is simple: a speech score should not stand alone. It should be supported by observable measurements, explainable evaluation rules, and evidence the speaker can inspect.

What it does

VoxTrace combines browser-based audio analysis, transcript-based evaluation, an interactive results dashboard, and a contrastive dataset experience.

1. Analyse speech locally

Users can upload or record audio and provide the corresponding transcript. Local Analysis processes supported audio in the browser using the Web Audio API. It estimates recording duration and identifies significant quiet intervals using waveform amplitude thresholds.

The transcript analysis uses deterministic rules to identify filler words, repetition patterns, transitions, hedging language, sentence characteristics, and evidence-related cues.

2. Evaluate five dimensions

VoxTrace calculates rubric-based scores across five dimensions:

  • Clarity: transcript-based indicators associated with understandable communication.
  • Structure: sentence and transition patterns associated with organisation.
  • Supporting evidence: language cues associated with examples and supporting material.
  • Fluency: filler words and repeated-word patterns.
  • Pacing: speaking-rate estimates and detected quiet intervals.

The local rubric uses fixed rules and scoring weights, making its calculations reproducible for the same inputs and configuration.

These scores are explicitly presented as heuristic indicators, not expert judgements or proof that an argument is factually correct.

3. Ground feedback in evidence

VoxTrace distinguishes between time-grounded audio findings and transcript-level observations.

When a finding is supported by a valid audio interval, users can navigate to that moment in the recording. When an observation cannot be reliably aligned with the audio, it is displayed as transcript-level feedback rather than being assigned an invented timestamp.

This distinction is central to the project: useful temporal feedback must be grounded in available evidence.

4. Compare ideal and flawed speeches

The Contrastive Dataset view presents paired ideal and flawed examples, labelled reference flaws, and side-by-side rubric comparisons.

The goal is to make differences in evaluation understandable and to provide a foundation for assessing speech feedback against reference annotations.

The current dataset examples are demonstration and reference fixtures; we do not claim a validated benchmark accuracy or model-training result.

How we built it

VoxTrace uses React, TypeScript, and Vite for the application, with the Web Audio API for browser-based audio analysis. Shared components provide score visualisations, rubric breakdowns, feedback cards, transcript views, audio controls, and contrastive comparisons.

Evaluation metadata is stored locally through the existing history mechanism. Raw audio binaries are not stored in localStorage.

The application includes an optional server-side OpenAI integration, but the primary Local Analysis workflow does not require an API key, paid inference, or remote AI requests. During development, live analysis testing was blocked by OpenAI rate-limit responses, so we focused on making the local workflow useful without pretending that failed API requests had succeeded.

AI coding assistance was used during development, with the implementation and output remaining the responsibility of the project team.

Challenges and what we learned

One major challenge was separating measurable audio observations from interpretations of the transcript. A quiet interval can be estimated from waveform amplitude, but it is not automatically a confirmed speech flaw. Likewise, a transition word may suggest structure without proving that an argument is well organised.

We addressed this by keeping the rubric fixed, using explicit heuristic rules, and separating timestamped findings from observations that cannot be reliably aligned to a moment in the audio.

A second challenge was building a useful system without depending on paid AI inference. Local processing allows the core analysis workflow to operate independently of API availability and makes its scoring rules easier to inspect.

The project also reinforced the importance of honest evaluation: seeded examples are not experimental evidence, heuristic scores are not expert judgements, and successful compilation does not replace testing with real recordings.

What's next

The next steps are end-to-end validation across real browser recordings, broader manually annotated ideal-versus-flawed speech pairs, and systematic evaluation of detection quality against reference labels.

VoxTrace aims to make speech feedback more explainable: not just a score, but an observation, its supporting evidence, and a practical next step.

Built With

Share this project:

Updates

Submission history