RhetorTrace

• 

  Overview of the RhetorTrace with key findings and a visual speech waveform.

• 

  Explains a specific flaw with its exact timestamp, acoustic evidence, reference comparison, and confidence.

• 

  Shows the complete speech timeline, detected flaws, score, and transcript with highlighted findings.

• 

  Detailed interactive view for exploring word-level timing, findings, pitch, energy, and reference differences.

The idea Most speech-analysis tools give users a score or generic feedback such as “speak slower” or “improve your pitch.” We felt that this misses the most useful part of feedback: where exactly did the problem happen, and what evidence supports it? That question inspired RhetorTrace, our submission for Track C — Contrastive Speech Analytics & Temporal Flaw Grounding of the Multimodal AI Hackathon 2026. Our core idea is to compare a participant's speech with reference recordings of the same text, rather than judging speech against arbitrary universal thresholds. We wanted the system to trace a delivery problem back to a specific moment in the recording and explain the deviation using measurable acoustic evidence.

How we built it RhetorTrace takes a participant's audio, transcript, and same-text reference recordings through a multi-stage analysis pipeline:

1. Word-level alignment maps the transcript to precise timestamps.
2. Acoustic feature extraction measures pitch/F0, energy, speech rate, pauses, and spectral features such as MFCCs.
3. Speaker-agnostic normalization makes comparisons more robust across different voices.
4. A contrastive baseline is constructed from the reference recordings.
5. Robust statistical deviation detection identifies suspicious temporal regions.
6. Findings are assigned to categories such as pacing, pitch variation, pauses, energy, and clarity.
7. The system produces an evidence-backed explanation for each finding. The result is an interactive timeline where users can move directly from a finding to the affected words, waveform region, acoustic evidence, and corresponding audio. A key design principle was that the language model should not invent the analysis. The underlying pipeline produces the measurements and evidence first; AI is used only to turn those structured findings into understandable feedback.

Building and testing the system We didn't want to evaluate RhetorTrace only on a few successful examples. We created a controlled contrastive dataset containing good references, deliberately introduced delivery flaws, multiple severity levels, and additional recordings for robustness testing. We tested the system across pacing, pitch, pauses, energy, and other controlled variations, including changes in volume, sampling rate, compression, and noise. On our evaluation set, RhetorTrace achieved an overall F1 score of 0.827, with 0.774 precision and 0.888 recall. We also evaluated temporal localization and severity monotonicity to determine whether the system was not only detecting flaws, but locating and ranking them meaningfully.

Challenges One of the biggest challenges was temporal precision. Detecting that a speaker has a pacing problem is relatively easy; determining exactly where that problem starts and ends is much harder. Another challenge was distinguishing genuine delivery differences from normal variation between speakers. This led us toward the contrastive, same-text reference approach and robust normalization rather than relying on fixed thresholds. We also encountered practical engineering challenges around long-running audio models, memory usage, reproducibility, and integrating a scientific analysis pipeline into a usable application. We eventually built a proper analysis server with isolated jobs, validation, status tracking, structured results, and a reproducible evaluation pipeline.

What we learned The biggest lesson from building RhetorTrace was that good AI feedback needs evidence and context. A prediction such as “your pacing is poor” is much less useful than: “This 4-second region is significantly faster than the reference, affecting these words.”

That shift — from scoring to tracing and explaining — became the central design philosophy of RhetorTrace. We also learned that building a multimodal AI system is not only about choosing powerful models. Dataset quality, temporal alignment, normalization, evaluation, robustness testing, and reliable engineering can matter just as much. Ultimately, RhetorTrace is our attempt to make speech analysis less of a black-box score and more of an interactive, measurable timeline of delivery.

Built With

• artificial-intelligence
• audio-processing
• data-visualization
• fastapi
• machine-learning
• parselmouth
• python
• pytorch
• react
• signal-processing
• speech-processing
• speech-recognition
• typescript
• web
• whisperx