-
-
Arachne
-
Hero
-
Pipeline
-
Dataset
-
Results
-
Stress Test
-
Conclusion
Inspiration
Judging interpretive, persuasive and extemporaneous speeches is subjective. Two judges can hear the same performance and give different scores, and the feedback is often vague ("pick up the pace", "more energy").
What stayed with us was credibility and accountability. A judge who can't say why a delivery was marked down is hard to trust, and a speaker who can't see where it slipped has nothing to practice. We also kept coming back to a distinction: being inspired by someone's speech is entirely different from copying it. Learning from a great delivery means understanding what made it work, not reproducing it beat for beat.
So Arachne uses a reference recording of the same words as a yardstick, not a template to imitate. It shows where a performance measurably differs, and it keeps the judgment with the person: every finding is evidence tied to a measurement and a rule, never a verdict.
What it does
Arachne takes a speech recording and its transcript, aligns the words to the audio, and compares the performance with a documented reference for the same text. It then:
- measures pacing, pause placement and duration, and a relative energy (or pitch) contour,
- marks time-bounded regions where the delivery deviates from the reference,
- maps each deviation to an explicit rubric rule and a score, and
- shows everything in an interactive dashboard: audio playback, transcript, time-series overlays, and an explanation card for each flag.
Every explanation names the measured feature, both values, the time span and the rubric rule behind it. A deviation is evidence for a rubric finding. It is not proof that the delivery was wrong or why it happened, and a delivery that differs from the reference is not automatically worse. Arachne returns "insufficient evidence" when the alignment, audio quality or reference coverage isn't good enough.
How we built it
Dataset. We started small: three public speeches, each with a documented reference recording and matched flawed versions of the exact same text across a few severity levels, from near-perfect to clearly flawed. Flawed versions come from controlled audio edits (Tier 1) and human re-recordings (Tier 2). Every item keeps its source, transcript, transformation settings and reviewed flaw spans together, and each source's rights were checked for both audio and text.
Features. Words are forced-aligned to the audio, then we extract word timings, pauses and energy in dB. To compare different speakers fairly, pitch is expressed relative to each speaker's own median:
$$\mathrm{st}(t) = 12\,\log_2!\left(\frac{f_0(t)}{\tilde f_0}\right)$$
Scoring. Scoring is deterministic and rule-based, with no language model in the scoring path. We compare each participant against the reference using robust z-scores, with the spread estimated from calibration pairs of near-perfect reads:
$$z = \frac{\Delta - \tilde{\Delta}{\mathrm{cal}}}{1.4826\cdot \mathrm{MAD}{\mathrm{cal}}}$$
Same input and same pinned configuration give the same output.
Evaluation. We compare predicted regions with reviewed ground-truth spans using intersection-over-union and start/end boundary error, and we report missed and false-positive regions:
$$\mathrm{IoU}=\frac{|P\cap G|}{|P\cup G|}$$
Results: [FILL: metrics, each with sample count and label tolerance, plus at least one failure or "insufficient evidence" example].
Dashboard. An interactive Streamlit app with Plotly time-series overlays, synchronized audio playback and transcript highlighting, and a limits panel that shows what is and isn't measured.
Tech stack
| Layer | What we used |
|---|---|
| Language and data | Python; WAV audio (16 kHz mono for analysis, originals kept); plain-text transcripts; JSON manifests and feature/score files with versioned schemas; YAML configuration with a config hash |
| Alignment | WhisperX forced alignment against the supplied transcript, with a flagged fallback when alignment fails |
| Features | Word durations and local speaking rate, pause intervals, energy contour in dB, voicing; F0 in semitones relative to the speaker's median and MFCC/spectral features where used [FILL: confirm which shipped]; numpy and scipy for processing; ffmpeg for decoding uploads [FILL: confirm] |
| Dataset tooling | Scripted, seeded audio edits for Tier 1 flaws (time, gain and silence edits); human re-recordings for Tier 2; reviewed span labels; public-domain and rights-checked sources such as the Miller Center collection [FILL: list the speeches actually used] |
| Scoring | Median and MAD robust z-scores; versioned rubric rules; template-based explanations; evaluation by IoU, boundary error, precision and recall, and severity ordering |
| App | React with Vite, Plotly via react-plotly.js for time-series overlays, wavesurfer.js for playback and flaw regions |
| Reproducibility | Pinned dependencies and configuration; repeat-run tests; fresh-clone verification [FILL: container, if shipped] |
| Delivery | GitHub repository and README; public dataset link (GitHub or Google Drive); six-page technical document; YouTube demo; Devpost |
| Design | Inter and Instrument Serif (italic) typography; a black, white and single-icy-accent visual system shared by the landing page spec and the dashboard theme; hand-coded HTML/CSS/vanilla JS for the landing page; Mermaid for architecture and workflow diagrams |
Challenges we ran into
- No natural spread for a z-score. With one reference per speech there is no distribution to scale against, so we estimated it from near-perfect re-reads instead of the participant's own deltas.
- Different speakers and microphones. Raw pitch and loudness differ between people and recordings, so we normalized relative to each speaker and recording.
- Rights and reuse. Each source recording and its transcript needed its own check, and anything uncertain stayed out of the dataset.
- Public audio isn't clean. Old or noisy recordings hurt alignment and pitch tracking, so we chose cleaner sources and flag low-confidence regions instead of hiding them.
- Reference is a yardstick, not a goal. Matching the reference is not the same as speaking well, so Arachne reports measured differences and never claims a deviation is objectively wrong.
- Honest claims. We avoid claiming millisecond accuracy, and we report small-sample results as a prototype evaluation.
- Twelve days, four people. We kept the scope small (three speeches, three delivery dimensions) and protected review quality and integration time.
What we learned
Defining the rubric before generating data made everything downstream clearer. Matched-text comparison removes a lot of noise. Explanations people can trust come from showing the measurement and the rule, not just a score. And a small, carefully reviewed dataset taught us more than a large messy one would have.
The team: Kelambakkam Atti
| Member | Role | Focus |
|---|---|---|
| Ammar | Data lead | Source speeches, rights checks, paired reference and flawed recordings, annotation and labels |
| Abdul | Signal lead | Alignment, preprocessing, normalization, feature extraction, repeatable pipeline |
| Rihan | Grounding and scoring lead | Rubric, deviation model, flaw-region detection, explanations, evaluation |
| Fadil | Dashboard and delivery lead | Dashboard, integration, README, technical document, demo video, Devpost submission |
Each of us owned our own part, and everybody also worked in every department to help the others out: recording and reviewing data, testing the pipeline, checking scoring output, and polishing the dashboard and submission. The project came together because everyone did their own part and then helped the rest finish theirs.
AI assistance
We used Cursor, Gemini, ChatGPT and Claude, all on free plans. AI-assisted parts of the project:
- Planning: breaking the project into workstreams, schedule and gates, and drafting the role process files.
- Feasibility checks: testing whether scope, tools and timelines were realistic for twelve days.
- Cross-validation: a second look at our rubric logic, scoring method, evaluation approach and submission checklist, and spotting gaps and contradictions in our plan.
- Design help: the visual design spec, copy, and the architecture and workflow diagrams.
- Coding assistance: [FILL: which code was AI-assisted, e.g. scaffolding, refactoring, debugging].
- Writing: drafts of documentation, the project story and the pitch, which the team edited.
Dataset recordings, label review, rubric decisions and the evaluation results are ours, and every AI suggestion was checked by a teammate before we used it.
What's next
More speakers and recording conditions, more delivery dimensions, a larger reviewed dataset, and a stronger calibration of the thresholds.
Log in or sign up for Devpost to join the conversation.