-
-
Describe it live: the full pipeline runs on real providers over server-sent events, reading from B2 and writing the results back.
-
A described lecture playing: it reads the equation aloud, shows the cost versus a human describer, and the four files written back to B2.
-
Toggle Original and the pause is silent; toggle Described and Overtone's audio description fills it. Same file, in place on your B2 bucket.
-
Overtone generates WCAG audio description for a whole video archive, in place on Backblaze B2. About $0.03 a minute versus $15-75 by hand.
-
The problem: in 2017 UC Berkeley removed 20,000+ lectures rather than remediate them. Overtone makes describing an archive cheap to do.
-
Real footage: an MIT OpenCourseWare chalkboard lecture, described as the professor writes, with automatic failover across providers.
-
The pipeline: read source video from B2, find the dialogue pauses, describe and fit each one, write the described master back to B2.
-
A small Python CLI runs against any B2 bucket at archive scale, describing every video not already done and resuming safely.
Inspiration
In 2017 UC Berkeley pulled more than twenty thousand lectures offline. Not because the content was wrong, but because making it accessible cost more than deleting it. That is the quiet failure mode of video accessibility: WCAG 2.1 (success criterion 1.2.5) requires a spoken audio-description track on prerecorded video, a narrator describing what is on screen in the pauses in dialogue, and almost nobody does it because captions are cheap and description is not.
Human describers charge $15 to $75 per finished minute. For an archive that is not expensive, it is impossible: 10,000 hours at even $7 a minute is over four million dollars. So archives stay non-compliant, or get taken down. Public US universities owe WCAG 2.1 AA under the DOJ's Title II rule (deadline April 26, 2027 for large entities), and the EU Accessibility Act has been binding since June 2025. The demand is real, funded, and legally forced. The tooling to serve it at archive scale is not.
What it does
Point Overtone at a Backblaze B2 bucket prefix. For every video that is not already described, it:
- reads the video out of B2,
- transcribes it with AssemblyAI to get word-level timings,
- finds the pauses a describer would speak into,
- samples keyframes, skipping slides that have not changed,
- describes each pause with a vision model, reading the board aloud: equations as spoken math ("y equals m x plus b"), diagrams traced, on-screen text read verbatim, not just "an equation is shown",
- fits the narration to the pause: it speaks the description, measures the real audio, and if it ran long, rewrites it shorter,
- mixes it into the pause with the program audio ducked underneath, and for a pause too short to fit, freezes the frame and plays the full description (WCAG 1.2.7 extended description),
- writes the described master, a WebVTT track, a transcript, and a hash-verified provenance manifest back into the same bucket, beside the original.
Nothing leaves your storage. The WebVTT it produces is exactly what Panopto and Kaltura ingest as an audio-description track, so the result drops into the platform a university already runs.
AI audio description is a real category (Verbit, Visonic, ViddyScribe). I am not claiming to invent it. What none of them do, and what this hackathon is about, is run where the archive already lives: they are all services you upload your archive to and pay egress on. Overtone brings the pipeline to the bucket.
How I built it
A Python package with a CLI (doctor, scan, describe, archive) and a FastAPI web app, hosted in a Docker container behind Traefik. Every generative step is a Genblaze provider; the connective logic that no single provider offers is Overtone's own and is the interesting part: dialogue-pause detection that merges overlapping speakers so it never invents a gap, a difference-hash change detector that avoids re-describing a static slide, the fit loop that derives its retry budget from the rate the voice actually delivered, and timeline-accurate mixing with audio ducking. About 150 tests, including ffmpeg integration and a fit loop proven to converge against a mock provider.
Challenges I ran into
Fitting narration to a pause was the hard one. Word counts predict spoken duration badly because every voice has its own pace, so the loop speaks the line, measures the rendered audio, and recomputes the budget from the rate that voice actually delivered. It converges in one or two tries instead of oscillating.
Not describing the same slide eight times mattered for cost. An average-perceptual-hash could not tell two equations apart (four bits different); a difference hash could (twenty-one bits). That switch is what makes archive-scale cost real.
Running it on a real MIT OpenCourseWare lecture surfaced two things a synthetic demo never would: real lectures have many short pauses, and firing that many vision calls quickly hits a provider's per-minute rate limit and kills the run. I added escalating-backoff retry with cross-provider failover so a big run survives.
And the canonical multimodal message worked on the OpenAI connector but was rejected by the Google one, so vision was not actually provider-swappable. I filed the bug and the maintainers merged the fix during the hackathon.
Accomplishments that I'm proud of
It runs live on real infrastructure, not in a simulator: a real Backblaze bucket, real providers, results you can play. Cost is measured, not projected: a real three-minute MIT lecture segment cost about nine cents to describe, roughly three cents a minute, and the descriptions read the board ("y equals m x plus b").
Two of the three pieces of feedback I filed on the Genblaze SDK were merged and shipped during the hackathon: the Gemini multimodal fix (issue #194, PR #217) and opt-in rate-limit backoff (issue #221, PR #229). The app now runs on the release that contains both, using the official path.
What I learned
The winning insight was about direction. Almost every entrant will read the prompt as prompt then generate then store, which makes B2 a dumb sink. Backblaze's real media business runs the other way: customers already keep petabytes of existing video in B2. Making B2 the source as well as the destination, reading dormant archive out and writing described masters back, is the un-obvious move, and it is the thing a storage company most wants told. I also learned how much a days-old SDK improves when you build hard against it and feed real bugs back.
What's next for Overtone
A human-review queue so an accessibility office can approve or edit a description before publish; a native Panopto and Kaltura integration so the track lands in the LMS automatically; and speaker-diarized description that names who is on screen.
Log in or sign up for Devpost to join the conversation.