-
-
Arachne
-
Hero
-
Pipeline
-
Dataset
-
Results
-
Stress Test
-
Conclusion
Inspiration
Judging interpretive, persuasive and extemporaneous speeches is subjective. Two judges can hear the same performance and give different scores, and the feedback is often vague ("pick up the pace", "more energy").
What stayed with us was credibility and accountability. A judge who can't say why a delivery was marked down is hard to trust, and a speaker who can't see where it slipped has nothing to practice. We also kept coming back to a distinction: being inspired by someone's speech is entirely different from copying it. Learning from a great delivery means understanding what made it work, not reproducing it beat for beat.
So Arachne uses a reference recording of the same words as a yardstick, not a template to imitate. It shows where a performance measurably differs, and it keeps the judgment with the person: every finding is evidence tied to a measurement and a rule, never a verdict.
What it does
Arachne takes a speech recording and its transcript, aligns the words to the audio, and compares the performance with a documented reference for the same text. It then:
- measures pacing, pause placement and duration, and a relative energy (or pitch) contour,
- marks time-bounded regions where the delivery deviates from the reference,
- maps each deviation to an explicit rubric rule and a score, and
- shows everything in an interactive dashboard: audio playback, transcript, time-series overlays, and an explanation card for each flag.
Every explanation names the measured feature, both values, the time span and the rubric rule behind it. A deviation is evidence for a rubric finding. It is not proof that the delivery was wrong or why it happened, and a delivery that differs from the reference is not automatically worse. Arachne returns "insufficient evidence" when the alignment, audio quality or reference coverage isn't good enough.
How we built it
Dataset. Three public-domain US speeches (Gettysburg Address, Second Inaugural, FDR First Inaugural). For each text we built 21 items: near-perfect mirrors, six single-factor flaw families (fast/slow pace, excess/missing pauses, flat pitch, quiet energy) at three severities, and a three-flaw combination. Audio is synthetic (espeak-ng) so ground-truth flaw spans are exact by construction. Each item keeps its source text, transformation settings and span labels. Real recordings and human-reviewed labels are the main next step.
Features. Words are forced-aligned to the audio, then we extract word timings, pauses and energy in dB. To compare different speakers fairly, pitch is expressed relative to each speaker's own median:
$$\mathrm{st}(t) = 12\,\log_2!\left(\frac{f_0(t)}{\tilde f_0}\right)$$
Scoring. Scoring is deterministic and rule-based, with no language model in the scoring path. We compare each participant against the reference using robust z-scores, with the spread estimated from calibration pairs of near-perfect reads:
$$z = \frac{\Delta - \tilde{\Delta}{\mathrm{cal}}}{1.4826\cdot \mathrm{MAD}{\mathrm{cal}}}$$
Same input and same pinned configuration give the same output.
Evaluation. We compare predicted regions with reviewed ground-truth spans using intersection-over-union and start/end boundary error, and we report missed and false-positive regions:
$$\mathrm{IoU}=\frac{|P\cap G|}{|P\cup G|}$$
Dashboard. An interactive React/Vite plus FastAPI app with Plotly time-series overlays, synchronized audio playback and transcript highlighting, and a limits panel that shows what is and isn't measured.
Tech stack
| Layer | What we used |
|---|---|
| Language and data | Python; WAV audio (16 kHz mono for analysis, originals kept); plain-text transcripts; JSON manifests and feature/score files with versioned schemas; YAML configuration with a config hash |
| Alignment | WhisperX forced alignment against the supplied transcript, with a flagged fallback when alignment fails |
| Features | Word durations and local speaking rate, pause intervals, energy contour in dB, voicing; F0 in semitones relative to the speaker's median and MFCC/spectral features where used; numpy and scipy for processing; ffmpeg for decoding uploads |
| Dataset tooling | Scripted, seeded audio edits for Tier 1 flaws (time, gain and silence edits); human re-recordings for Tier 2; reviewed span labels; public-domain and rights-checked sources such as the Miller Center collection |
| Scoring | Median and MAD robust z-scores; versioned rubric rules; template-based explanations; evaluation by IoU, boundary error, precision and recall, and severity ordering |
| App | React with Vite, Plotly via react-plotly.js for time-series overlays, wavesurfer.js for playback and flaw regions |
| Reproducibility | Pinned dependencies and configuration; repeat-run tests; fresh-clone verification |
| Delivery | GitHub repository and README; public dataset link (GitHub or Google Drive); six-page technical document; YouTube demo; Devpost |
| Design | Inter and Instrument Serif (italic) typography; a black, white and single-icy-accent visual system shared by the landing page spec and the dashboard theme; hand-coded HTML/CSS/vanilla JS for the landing page; Mermaid for architecture and workflow diagrams |
Challenges we ran into
- No natural spread for a z-score. With one reference per speech there is no distribution to scale against, so we estimated it from near-perfect re-reads instead of the participant's own deltas.
- Different speakers and microphones. Raw pitch and loudness differ between people and recordings, so we normalized relative to each speaker and recording.
- Rights and reuse. Each source recording and its transcript needed its own check, and anything uncertain stayed out of the dataset.
- Public audio isn't clean. Old or noisy recordings hurt alignment and pitch tracking, so we chose cleaner sources and flag low-confidence regions instead of hiding them.
- Reference is a yardstick, not a goal. Matching the reference is not the same as speaking well, so Arachne reports measured differences and never claims a deviation is objectively wrong.
- Honest claims. We avoid claiming millisecond accuracy, and we report small-sample results as a prototype evaluation.
- Twelve days, four people. We kept the scope small (three speeches, three delivery dimensions) and protected review quality and integration time.
Results
Setup. Synthetic dev dataset (63 items, 3 speeches × 21) with ground-truth spans that are exact by construction, not human-reviewed. Predicted regions are matched one-to-one to ground-truth spans of the same dimension by IoU (greedy, highest IoU first). We report recall at IoU 0.3 and 0.5, mean IoU of matches, mean absolute start/end error, false-positive regions, flags on near-perfect mirrors, and whether the dimension score falls as severity rises. Thresholds were set on one dev speech (Gettysburg). Second Inaugural and FDR 1933 were held out and not used for tuning.
Held-out: 2 speeches, 42 items
| Dimension | GT spans | [email protected] | [email protected] | Mean IoU | Start err (s) | End err (s) |
|---|---|---|---|---|---|---|
| Pacing | 14 | 0.857 | 0.857 | 0.919 | 0.174 | 0.113 |
| Pauses | 20 | 1.000 | 1.000 | 0.952 | 0.012 | 0.012 |
| Pitch contour | 8 | 0.875 | 0.875 | 0.663 | 0.808 | 0.645 |
| Energy | 6 | 1.000 | 0.833 | 0.886 | 0.284 | 0.239 |
4 false-positive regions. 0 flags on 4 near-perfect mirrors. Severity ordering correct in 12/12 series.
All 63 items
| Dimension | GT spans | [email protected] | [email protected] | Mean IoU |
|---|---|---|---|---|
| Pacing | 21 | 0.905 | 0.857 | 0.894 |
| Pauses | 30 | 1.000 | 1.000 | 0.951 |
| Pitch contour | 12 | 0.833 | 0.833 | 0.657 |
| Energy | 9 | 1.000 | 0.889 | 0.921 |
5 false-positive regions. 0 flags on 6 near-perfect mirrors. Severity ordering correct in 18/18 series.
Stress test (Gettysburg, 21 items, each recording perturbed and re-analyzed)
| Variant | FP regions | [email protected] (pace / pause / pitch / energy) |
|---|---|---|
| Clean | 1 | 1.00 / 1.00 / 0.75 / 1.00 |
| MP3 48 kbps → 44.1 kHz | 1 | 1.00 / 1.00 / 0.75 / 1.00 |
| +3 s lead/trail silence | 1 | 1.00 / 1.00 / 0.75 / 1.00 |
| Noise 30 dB SNR | 5 | 1.00 / 0.80 / 0.75 / 1.00 |
| Noise 20 dB SNR | 42 | 0.57 / 0.40 / 0.75 / 0.67 |
| Noise 10 dB SNR | 27 | 0.71 / 0.60 / 1.00 / 0.67 |
Failures and limits (reported, not hidden)
- Pause and energy flaws are located almost exactly. Pitch boundaries are loose (about 0.7 s) because the detector uses a 5-word window.
- The mildest pitch flattening (60% compression) was missed on the dev speech.
- Results survive MP3 compression, resampling and added silence, but not additive noise: at 20 dB SNR, false positives rise from 1 to 42 and pause recall falls from 1.00 to 0.40.
- Our SNR estimate only warned near 10 dB and never returned "insufficient evidence", so noisy recordings still received scores. This is our top planned fix.
- With 21 items per run, differences between noise levels (e.g. 10 dB looking better than 20 dB) are measurement noise, not a trend.
- These numbers show pipeline behaviour on controlled, noise-free, single-factor synthetic audio. They are not accuracy on human speech, and real recordings will be harder.
Reproduce with make evaluate and make stress. Raw outputs are in results/evaluation.json and results/stress.md.
What we learned
Defining the rubric before generating data made everything downstream clearer. Matched-text comparison removes a lot of noise. Explanations people can trust come from showing the measurement and the rule, not just a score. And a small, carefully reviewed dataset taught us more than a large messy one would have.
The team: Kelambakkam Atti
| Member | Role | Focus |
|---|---|---|
| Ammar | Data lead | Source speeches, rights checks, paired reference and flawed recordings, annotation and labels |
| Abdul | Signal lead | Alignment, preprocessing, normalization, feature extraction, repeatable pipeline |
| Rihan | Grounding and scoring lead | Rubric, deviation model, flaw-region detection, explanations, evaluation |
| Fadil | Dashboard and delivery lead | Dashboard, integration, README, technical document, demo video, Devpost submission |
Each of us owned our own part, and everybody also worked in every department to help the others out: recording and reviewing data, testing the pipeline, checking scoring output, and polishing the dashboard and submission. The project came together because everyone did their own part and then helped the rest finish theirs.
AI assistance
We used Cursor, Gemini, ChatGPT and Claude, all on free plans. AI-assisted parts of the project:
- Planning: breaking the project into workstreams, schedule and gates, and drafting the role process files.
- Feasibility checks: testing whether scope, tools and timelines were realistic for twelve days.
- Cross-validation: a second look at our rubric logic, scoring method, evaluation approach and submission checklist, and spotting gaps and contradictions in our plan.
- Design help: the visual design spec, copy, and the architecture and workflow diagrams.
- Coding assistance: things like debugging and documentation drafts were done using Claude, Cursor, Gemini and ChatGPT. All AI-written code was run and reviewed by a teammate.
- Writing: drafts of documentation, the project story and the pitch, which the team edited.
Dataset recordings, label review, rubric decisions and the evaluation results are ours, and every AI suggestion was checked by a teammate before we used it.
What's next
More speakers and recording conditions, more delivery dimensions, a larger reviewed dataset, and a stronger calibration of the thresholds.
