Heard

Inspiration

Turn on a screen reader and tab through almost any website, and you'll hear things like "button. button. Edit, button. Edit, button. Edit, button." The code is valid, so linters pass it, but a blind user has no idea what any of those buttons do.

Today, teams choose between automated linters, which are fast but only check rules, and accessibility experts who test with a real screen reader, which is thorough but slow and expensive, so most teams do it rarely or never. We wanted to automate the expert: something that listens to a product the way a blind user experiences it, then fixes what's broken and proves the fix works.

What it does

Heard takes a screen-reader session (the screen recording, the announcements, and the order of keyboard steps) and finds the places where what the user heard doesn't match what was on screen:

• a cart icon announced as just "button"
• a button that says "Create account" but is announced as "Continue"
• a validation error that appears but is never spoken
• a fake checkbox with no checked state

For each issue, Heard:

1. Listens: NVIDIA Nemotron 3 Nano Omni watches the video and hears the screen reader's audio in the same input.
2. Grounds: a deterministic check throws out any finding whose quote was never actually heard, and a live search restricted to w3.org cites the relevant WCAG rule.
3. Patches: Nemotron Super writes a minimal source fix. If it fails, Super gets one more try, then Nemotron Ultra.
4. Verifies: no model here. The patched page loads in Chromium, the accessibility tree must show the fix, and axe-core must report no new violations.

You can press Before and After to hear the difference.

For teams, Heard is also a CLI that runs on every push:

• heard check records what users hear, diffs it against the main branch, and sends only what changed to the model, so a push that doesn't change what users hear costs zero tokens.
• heard fix patches your source and keeps only fixes verified on your running app.

Results: on 12 seeded bugs across three sites, axe-core caught 4. Heard caught 10 and verified all 10 fixes, for $0.02 per full run.

How we built it

• Models:
  • Perception: Nemotron 3 Nano Omni on NVIDIA's API.
  • Patching and escalation: Nemotron 3 Super and Ultra on Nebius Token Factory.
  • Each model role can be pointed at either provider, with retries for busy endpoints and a cap on Omni's thinking tokens.
• Citations: Tavily search limited to w3.org, with a fixed WCAG rule per issue type, cached for 30 days.
• Verification: Playwright, the Chromium accessibility tree (via the DevTools protocol), and axe-core. Each issue type has its own check: accessible name, label in name, live region, dialog focus, tab state, heading order.
• Sessions: a simulated screen reader built from the browser's accessibility tree, spoken with text-to-speech and combined with the screen recording.
• App: a FastAPI backend that streams progress to a React/Tailwind dashboard with light and dark mode. The UI passes axe with zero violations, which a test enforces.
• CLI: a Python package (uvx heard) with an npm launcher (npx heard). check writes a self-contained HTML report, and there's an example GitHub Action.
• Deployment: Docker on Render.

Challenges we ran into

• Token Factory doesn't accept audio. Its chat API takes only text, image and video. Sending audio is rejected (422) for every model, and there's no transcription endpoint. Since listening is the whole point, we tested every route and moved perception to Nemotron Omni on NVIDIA's API, which hears a video's own soundtrack.
• Models invent quotes, and they also format real ones creatively. Our evidence check kept rejecting real bugs, because the model quoted two announcements together or wrote silence as "(silence)". We made it tolerant of formatting while still rejecting quotes that were never heard.
• Honest scoring. Our first benchmark said "9/9 fixes verified". Digging in, one bug was credited with a fix for a different issue on the same element. We tightened the rule to an honest 8/9, then made the fixes that earned the 10/10.
• Reasoning models. Thinking tokens count against the output limit, which produced empty answers. Very long thinking also caused gateway timeouts. Capping Omni's thinking cut perception time by more than half with the same accuracy.
• One parsing bug cost a whole fix. Ultra once returned a correct patch followed by extra text, and our parser discarded it. Reading only the first complete JSON value fixed it.
• One bug is inaudible. A form field labelled only by its placeholder sounds fine, because the browser names the field from the placeholder. We kept it in the benchmark and say so.

Accomplishments that we're proud of

• 10/12 vs 4/12 against the industry-standard linter, with every detected fix verified by a browser rather than a model, at about two cents per run.
• Verification that can say no. Every reference fix passes, every unpatched control fails, and fixes that add new violations are rejected.
• Guards against false positives:
  • findings whose quote was never heard are thrown out
  • mistyped findings are corrected
  • findings the unpatched page already passes are rejected as "not reproduced", so they can never count as fixes
• A CI tool that's cheap enough to run on every push. Unchanged speech costs zero tokens, and the committed snapshot lets reviewers see what screen-reader users will hear differently.
• The tool for accessibility is itself accessible: zero axe violations in light and dark mode, enforced by tests.

What we learned

• The biggest accessibility gaps are about meaning, not markup. Linters verify that attributes exist. Users need them to make sense.
• The pattern that made results trustworthy: models find and fix, deterministic code decides.
• Probe capabilities, don't assume them. "HTTP 200" doesn't mean the model understood the input, so our probe requires a correct answer about a red image saying "HEARD", a video that turns blue then green, and a spoken "pineapple seventy three".
• Measure honestly. The most useful moments were when the numbers looked too good and we checked why.

What's next for Heard

• Real screen readers: run VoiceOver and NVDA recordings through the same pipeline, with Omni transcribing sessions that have no speech log.
• Component frameworks: a source locator that maps a broken element to the right React, Vue or Svelte component, and verifies against the dev server.
• More bug classes and real sites, plus multi-step flows like checkout and sign-in.
• Publish the heard CLI on PyPI and npm, and ship a one-click GitHub Action that comments on PRs with what users will hear differently.

Built With

• axe-core
• chromium
• docker
• fastapi
• github-actions
• httpx
• nebius-token-factory
• nemotron-3-nano-omni
• nemotron-3-super
• nemotron-3-ultra
• npm
• nvidia-api
• nvidia-nemotron
• playwright
• pydantic
• python
• react
• render
• tailwindcss
• tavily
• typescript
• uv
• uvicorn
• vite
• wai-aria