Unsaid

Lecturers say “as you can see” and blind students can’t. Unsaid finds those moments and describes them in the pauses, never over the lecturer, using NVIDIA Nemotron models.

Comment

Inspiration

Watch a lecture with your eyes closed. “…and as you can see, this curve tells us everything.” Which curve? Lecturers point instead of speaking all the time, and in a study of 90 presentation videos, 72% of 610 visual elements were insufficiently described by the speaker (Peng et al., CHI 2021). For a blind or low-vision student, every one of those moments is a gap in the lesson. Audio description fills such gaps, and WCAG 2.1 AA requires it for prerecorded video: US public universities must comply from April 2027. But description is slow and expensive, and existing AI tools describe scenes, not the moment a lecturer says “this”.

What it does

Unsaid takes a lecture video and returns an audio-description track for what the picture shows but the speech never says:

  • It finds pointing moments: “as you can see”, “these points”, “over here”.
  • It describes what they point at in the lecturer's pauses, pauses the video when no pause is long enough, and never speaks over the lecturer.
  • Every description is checked by code and re-checked against the frames, and shows the evidence.
  • It exports a WebVTT description track, captions, a transcript with descriptions inline, and Final Cut Pro markers.
  • It suggests the sentence the lecturer could say next time, so the fix happens at the source.
  • Long lectures go to an overnight batch job (built for Nebius Serverless Jobs) with a dollar cap.

Try the 30-second test on the home page: hear a clip as a blind student does, then hear it with Unsaid.

How we built it

Nine stages, four Nemotron models, each doing the one job it is best at. Unsaid is written for Nebius Token Factory, but every run shown here used NVIDIA's API catalog instead (see Challenges):

  1. Listen: NVIDIA Parakeet TDT 0.6B v2 (ONNX, CPU) transcribes with word timings and finds every silence. Super VL has no audio input, so this is how Unsaid knows when it may speak.
  2. Segment: code splits the video at slide changes and flags pointing phrases (“as you can see”, “this curve”).
  3. Watch: Nemotron 3.5 Super VL reads each segment as a low-frame-rate clip plus full-resolution stills, and returns strict JSON: slide text, charts with values, actions with times.
  4. Resolve: Nemotron 3 Nano decides what each “this” or “here” actually points at. It rejects figures of speech, hypotheticals and phrases about time. (Our NVIDIA API catalog key had no Nano, so in the runs shown Nemotron 3.5 Lightning does this step.)
  5. Context: Tavily reads the video’s own page and any works cited on slides, so names come from sources and never from faces.
  6. Understand: Nemotron 3 Ultra reads the whole lecture in one context: the transcript, every segment report and every silence with its word budget. It decides what is visual-only and how important it is, and drafts each description to fit its pause.
  7. Write & fit: code gates decide. A description must fit its silence (planned at 2.5 words per second, with guard bands) and must never overlap speech. It may use no number that isn’t printed on screen, no name without a source, and no “we see”. Nemotron 3.5 Lightning repairs failures; code trims; only essential facts may pause the video.
  8. Check: Super VL judges every claim against frames from that moment (supported, contradicted or not visible), which sets each description’s tier: ready, needs review or withheld.
  9. Deliver: tracks, transcript, markers and lecturer tips.

Every model call goes into a ledger with tokens, cached tokens, reasoning tokens, latency and cost at Token Factory list prices. That gives each video a price ($0.021 per minute of video in the evaluation) and a counterfactual: the same text work on Ultra alone would cost 60% more.

Results

On 9 openly licensed lecture clips (16.6 minutes, 38 labelled pointing moments), each described twice, measured on NVIDIA's API catalog (build.nvidia.com) with the same Nemotron models (Lightning standing in for Nano, which our catalog key lacked), at no cost to us, with costs priced at Token Factory list prices, compared with the obvious one-shot approach (Super VL given the whole clip and the timed transcript, asked to place descriptions in pauses):

  • Spoken over the lecturer: Unsaid 0.0 s per minute of video; one-shot 28.6 s.
  • Descriptions that fit in a pause: Unsaid 100%; one-shot 0%.
  • Pointing moments covered: Unsaid 46%; one-shot 40%.
  • Descriptions with an invented number or name: Unsaid 0%; one-shot 12%.
  • Claims the frames do not support: Unsaid 26%; one-shot 22%.
  • Video pauses for extended descriptions: Unsaid 1.99 per minute; one-shot 0.00.
  • Model cost per minute of video: Unsaid $0.021; one-shot $0.0075.

Unsaid covered more of the pointing moments (46% against 40%) without speaking over the lecturer. Unsaid has more claims the frames do not support (26% against 22%), although that row favours it: the same Super VL verifier scores both, and Unsaid's own check step answers to it. 19 of its 20 unsupported claims are ones the checker could not see in its frames rather than contradictions.

Six of the clips were also used to develop Unsaid's prompts. The other three were labelled blind and never used to tune them (Unsaid ran on them once, before one later change to where paused descriptions start), and their labels include pointing moments our phrase detector misses. On the held-out clips alone, Unsaid covered 38% of the pointing moments (one-shot 41%) and spoke over the lecturer for 0.0 s per minute (one-shot 22.7 s). The Evaluation page reports both sets separately and lists where Unsaid still fails. No phrase from any clip appears in a prompt.

Challenges we ran into

  • No Token Factory credit. The hackathon's $25 credit form rejected the event code in the official Devpost link (“the event in your request was incorrect”). Nebius handles such cases only on Discord, which is not accessible from Turkey, and we chose not to pay for calls. So Unsaid, written for Token Factory, made every call shown here through NVIDIA's API catalog, which serves the same Nemotron models through the same OpenAI-compatible API. This means the entry has not made the runtime Token Factory call the rules ask for; we would rather say so than hide it.
  • Super VL has no audio. We added NVIDIA Parakeet on CPU to know when the lecturer speaks; descriptions are timed against its speech intervals.
  • Fitting a description into 2.4 seconds. A free-writing model overshoots. Moving the draft into Ultra’s plan, where every silence has a word budget, cut extended (pausing) descriptions from 6 of 8 to 2 of 7 on our first test lecture.
  • Thinking models and budgets. Nemotron thinks by default. A small budget returns empty content, and long non-streamed reasoning calls hit gateway timeouts. We switch thinking off where it isn’t needed, give Ultra an explicit budget, and stream it.
  • Structured output that sometimes isn’t. We send the schema in the prompt as well as through constrained decoding, validate it with pydantic, and repair or retry. Truncated JSON gets more room instead of a “repair”.
  • Counting, and a checker that could only delete. Super VL can read twenty sampled points as “bars 0 through 10 and 11 through 20”, and Ultra turns that into “11 at 10, 9 at zero” (the frame shows ten and ten). On one run the frame check caught it, but the corrector was never shown what the checker saw, so all it could do was delete: the description became “rock bars adjusted”. Passing the observation through fixed that, and a correction may no longer shrink a description to a fragment. On another run the check passed the wrong count. Code cannot catch a plausible count, so the Evaluation page lists such failures and tiers mark what a person should check.
  • Requests that never answer. A few requests hung until the client gave up. Every call now streams, so a stall is cut off after a minute without a chunk instead of after a three-minute budget.
  • “This” is not always pointing. “In the right polarity”, “at this point” and “students could look at the graph later” all match a naive lexicon; Nano filters them.

Accomplishments that we're proud of

  • Descriptions never talk over the lecturer: timing is enforced in code, not hoped for.
  • Every description carries its evidence, and the interface itself passes automated WCAG 2.1 AA checks (axe) on every page, works by keyboard, and reflows on a 320-pixel screen.
  • A complete product: try it live, review, edit with the same checks, export, and run overnight.

What we learned

The most valuable model call is often the one that holds the whole problem at once. Ultra planning the lecture as a whole beat any number of local rewrites. Cheap models are best at repair, not authorship. And the best accessibility fix happens before the recording, which is why Unsaid coaches the lecturer.

What's next

  • Reviewer edits become cached examples for the planner and feed a data flywheel.
  • More languages, as Nemotron adds them.
  • An LMS integration (Canvas, Moodle) so the backlog job runs on every new upload.
  • A pilot with a university accessibility office before the April 2027 deadline.

Built With

  • axe-core
  • docker
  • fastapi
  • ffmpeg
  • github
  • nemotron-3-ultra
  • nemotron-3.5-lightning
  • nemotron-3.5-super-vl
  • nvidia-api-catalog
  • nvidia-nemotron
  • nvidia-parakeet
  • onnxruntime
  • playwright
  • pydantic
  • python
  • react
  • tavily
  • typescript
  • vite
Share this project:

Updates

Submission history