interlude

You captioned every lecture. Your blind students still can't follow one.

Comment

Inspiration

You record a lecture for your university, add captions, and post it online. You think you've made it work for every student. You just missed your blind students.

Captions are for deaf students. They turn speech into text. But your professor spends half the class at the board, drawing a diagram, circling what matters, saying nothing while they do it. There's no speech to caption in those silences. A blind student plays the recording and hears only the gap, right where the lesson is.

What they need is the opposite of a caption: the things that happen on screen while nobody is talking. That has a name, audio description, and there's a whole industry that makes it by hand, priced per minute.

And this isn't optional anymore. ADA Title II's 2024 rule requires US public universities to meet WCAG 2.1 AA, and that standard requires audio description for prerecorded video, with compliance deadlines in 2026 and 2027. A captioned-only lecture library doesn't meet it, and the blind students it leaves out are the reason.

We wanted to see if we could generate that description for a whole video library, automatically.

We called it Interlude because that's where it lives. The pause between the acts, the moment the talking stops. That's the exact space this works in: the silences between a lecturer's words. The narration is the interlude.

What it does

You give Interlude a lecture. It finds the silences between the lecturer's words, works out which ones actually hide something a blind listener would miss, writes narration short enough to fit the gap, and hands back the same lecture with a described audio track. No transcript to read, no extra file to process. Just a video whose audio now carries what the board was showing.

The important word is works out. Most silences should stay silent. Narrating a lecturer just walking across the room makes the track worse, not better. Over-narration is the number one way automated description fails, so Interlude leaves most gaps alone on purpose, and states why for each one.

On a 17-minute MIT physics lecture, it found 16 silences and narrated only 2. The other 14 each got a reason: "the speaker writes a formula that he reads aloud as he writes it," "the speaker is simply walking across the front of the lecture hall." A tool that narrates 2 of 16 and can defend the 14 it skipped is doing the right thing. A tool that narrates all 16 is the failure.

The part we're proudest of: it grades itself like a blind listener would

Anyone can get a model to produce descriptions. The hard part is knowing if they're any good, and the honest test is whether a blind person could rebuild the scene from the audio alone. We couldn't put blind listeners in a room this week, so we built the next best thing, and we run it on every clip.

A vision model watches the video and lists the key visual facts in each gap. After the descriptions are written and voiced, a second model, from a different provider on purpose (grading your own work with the model that did it is worthless), listens to the described audio with no video and reports which facts it can recover. Google Gemini writes; Groq's Llama checks. On the 2:55 linear algebra clip, 2 of 3 key facts were recoverable from the original audio, and all 3 after description. Small numbers, but honest ones. Measured, not guessed.

Coverage check and rewrite loop: facts recoverable audio-only, silences left silent with stated reasons, canonical hash and audit record

There's a second loop, and it's just arithmetic: the narration has to physically fit the silence. Interlude measures the real length of the rendered audio (with ffprobe, not a word-count guess), and rewrites shorter and re-renders if it runs long. We tuned the word budget to the voice's real speaking rate, so not one description ran past its gap in any final track.

How we built it

  • Storage and serving, Backblaze B2. Every pipeline stage writes its state to B2 (transcript, gaps, fill/skip decisions, description attempts, coverage check, final track), so a killed worker resumes exactly where it stopped. Uploads go straight from the browser to B2 by presigned PUT, and playback streams straight from B2 by presigned URL, so the server sits outside the hot path entirely. The audit trail lives in a second bucket with Object Lock, separate from the working bucket, because old attempts auto-expire and the audit record can't. Swap B2 for a local folder and the product stops working: the browser streams straight from these URLs, and the audit record loses its Object Lock guarantee. Storage isn't behind the product here, it's the product surface.
  • Text to speech, Genblaze Pipeline with ElevenLabsTTSProvider, writing to B2 through Genblaze's ObjectStorageSink. Its Manifest is our audit record: a hash-verified log of provider, model, prompt, parameters, and timing, locked into the compliance bucket. It's the one true media-generation step in the pipeline, so it's the one that runs through Genblaze. Transcription (Groq Whisper), vision and description writing (Gemini), and the coverage checker (Groq Llama 3.3 70B) are reasoning calls, not generation, so they run directly against their providers.
  • FastAPI and SSE stream pipeline progress live to the browser. React and Bun on Vercel, FastAPI on Railway.

Results

Real MIT OpenCourseWare lectures (CC BY-NC-SA), all live on the site with no login, two of them running the full pipeline in one click. Play the 2:55 linear algebra clip and hit the Original toggle. That's silence where the professor is drawing. Switch back and the board is narrated into the gap. That before and after is the whole product in ten seconds.

2:55 linear algebra clip with Original/Described toggle, 3 of 5 silences narrated, 3 of 3 facts recoverable

Across the three test lectures (17:20 physics, 2:55 linear algebra, 4:17 entrepreneurship) it found 16, 5, and 5 silences and narrated 2, 3, and 1 of them, leaving the rest silent on purpose, each with a stated reason. Not one description ran past its gap.

4:17 entrepreneurship lecture, 1 of 5 narrated, live narration filling a 5.7s silence

Challenges we ran into

A Genblaze version trap: the provider adapters ask for genblaze-core>=0.3.4 but import a function that only exists from 0.3.7, so 0.3.6 passes the check and still fails to import. Pinning to 0.3.8 fixed it. The audit file nearly got locked in the wrong place too. ObjectStorageSink.write_run copies every referenced asset into its own backend before writing the manifest, which would have copied our audio straight into the Object Lock bucket. We record the output as step metadata with its sha256 instead, so it stays tamper-evident without getting locked somewhere we didn't intend.

480p was too blurry for the board. Gemini's vision step is capped around 18MB, so we shrink to a silent 720p 2fps copy first. At 480p, handwritten equations went illegible and the model started skipping gaps it should have described. 720p fixed it. And ffmpeg runs as a subprocess rather than a library, because PyAV's unsigned DLLs get blocked by Windows Smart App Control on the dev machine. Transcription runs on hosted Groq Whisper for the same reason.

What we learned

Restraint is a feature, and it has to be measured, not claimed. The moment we could show which silences got left alone and why, the thing stopped looking like a description generator and started looking like one with judgement. The self-grading loop is what makes that claim checkable instead of a promise.

Worth being upfront about the edges. This doesn't replace what blind students already use for dense material. A long proof or heavy notation is still better served by MathML or a tactile diagram; description just adds to that. The coverage sample is small (1 to 3 key facts per short clip), so read those as honest counts, not a big percentage. And descriptions aren't human-reviewed yet. Production wants a person in the loop before anything publishes.

What's next

Gemini's Files API to lift the 15-minute analysis ceiling. The same fit engine that keeps narration inside a silence is what dubbing needs to keep translated audio in sync, so the pipeline extends naturally to serving a course in more than one language. And human review before publishing, for institutions that need sign-off.

Providers and models

  • Google Gemini: vision (which gaps to fill, and the key visual facts) and description writing
  • ElevenLabs: text to speech, run through Genblaze
  • Groq Whisper large-v3-turbo: speech to text
  • Groq Llama 3.3 70B: audio-only coverage checker, a different provider from the writer on purpose
  • Backblaze B2: all storage, with Object Lock on the provenance bucket
  • Genblaze SDK: runs the TTS generation and writes the provenance manifest

Built With

Share this project:

Updates

Submission history