Live site: https://lull-omega.vercel.app · Code: https://github.com/RaYYeR220/lull · Judging guide (5 minutes): https://github.com/RaYYeR220/lull/blob/main/JUDGES.md · Our demo video, audio-described by Lull: https://youtu.be/XlGzhnaBFLQ

Inspiration

Blind and low-vision viewers miss what is only on screen: the chart the lecturer never reads out, the sign in the shot, who just walked in. Audio description fixes that, and accessibility law now expects it for prerecorded video (WCAG 2.x success criterion 1.2.5; the EU Accessibility Act in force since June 2025; the US ADA Title II web rule). Human description is slow and expensive, so most video ships without it. The AI tools we tried had two habits that make description worse than nothing: they talk over the speaker, and they describe things that are not there.

What it does

Lull takes a prerecorded video and produces an audio-description track that is spoken only in the pauses between dialogue, plus WebVTT description tracks, an extended version that pauses the picture where a pause is too short, and a conformance report that says exactly what was checked and what still needs a person.

It listens first (NVIDIA Parakeet word timestamps + Nemotron diarization), so it knows every pause to the millisecond.
It watches (Nemotron 3.5 Super VL over the video), plans what matters with Nemotron 3 Ultra, and places descriptions with a constraint solver (CP-SAT) so nothing is ever scheduled over speech or after the visual it describes.
Before a line is voiced, the engine splits it into claims and whole sentences and has the vision model judge each against the frames. Anything unconfirmed is cut deterministically or the line is held for a person. Nothing is voiced unless its frame check passes.
It voices lines with NVIDIA Magpie TTS, measures the real audio, and refuses to publish if anything overlaps.
A person reviews in the Studio: edit, approve, hold, publish. The report only says "reviewed" when a person did.
Integrate it: an accessible <lull-player> web component (four listening modes), a REST API, a CLI, and an MCP server so agents can request descriptions.

How we built it

Models: all hosted on Nebius Token Factory — Nemotron 3.5 Super VL (watching, frame verification), Nemotron 3 Ultra (planning), Nemotron 3.5 Lightning (claim splitting, compression, repair) — plus NVIDIA speech models running in our own container via NeMo-Speech: Parakeet TDT 0.6B v3 (ASR), Nemotron diarization, Magpie TTS. Tavily web search runs in the research stage (names, places, works) when the video gives a reason.
Engine: Python, FastAPI, OR-Tools CP-SAT for placement, ffmpeg with strict input whitelists, a spend ledger with hard caps on every call.
Hosting: Google Cloud Run (one container: API + speech models on CPU), Vercel for the site. Total model spend for the whole build and evaluation: about $6.
Evaluation, pre-registered: answer keys and scoring rules were committed and tagged before anything was scored (tags in the repo), and every number on the site is generated from that.

Challenges we ran into

Making "it checked the frames" true by construction: early versions let a reworded false claim slip back in after a cut. The final design numbers the words of each line, cuts failed claims itself, and lets the language model only repair grammar under a check that the cut words cannot return.
Speech detection that was a few hundred milliseconds too generous swallowed most lecture pauses; the extended track first paused on the previous slide. Both were found by our own evaluation and fixed before the scored runs.
Cloud Run CPU makes text-to-speech slow; live runs take about six minutes for a 45-second clip.

Accomplishments that we're proud of

0.0 seconds of description over speech across every scored video.
An evaluation that reports its own weaknesses: on synthetic lectures Lull described 2.6 % of the hidden visual facts in the standard track (14.0 % with the extended track); against human describers it conveys little of what they chose to say (1–5 % of their facts, against 39–43 % between two people); and switching the frame check off did not measurably change unsupported statements while it raised recall. We publish all of it, with intervals.
A Studio that makes the human review fast, and a player any site can embed with one tag.

What we learned

Being careful is not the same as being useful: a describer that never interrupts and refuses to guess also says very little. The pre-registered evaluation told us that clearly, and the honest number is more valuable than a flattering one.

What's next

Raise coverage without giving up the two guarantees (denser descriptions in long silences, better use of the extended track), real classroom lectures with educators reviewing in the Studio, more languages, and GPU inference for speech so live runs take seconds, not minutes.