Unheard

The real NVDA screen reader inside GitLab CI: an accessibility fix fails until NVDA is heard saying the right words.

Comment

Inspiration

Many blind people use the web with NVDA, a free screen reader that reads each part of a page out loud. When a component is built wrong, NVDA says the wrong thing, and automated tests rarely notice: they check the code, not what the user hears. AI agents now write many of these fixes, and they can't hear either. Material UI has had a report open since June 2024: NVDA reads a five-star rating as "2 of 6".

What it does

Unheard puts the real NVDA in the GitLab pipeline.

  1. Plan. A nightly pipeline listens to watched Material UI reports with NVDA and opens a GitLab issue that records exactly what NVDA says today.
  2. Create. A person assigns the issue to the Unheard flow on GitLab Duo Agent Platform. One agent writes a "journey" (a listening test: the keys to press, the words NVDA must say and must never say). A second agent fixes the component and opens a merge request.
  3. Verify. The merge request pipeline starts a Windows VM in Google Cloud and plays the journey through NVDA in Chrome and Firefox, on the parent commit (it must hear the bug) and on the branch (it must hear the fix). NVDA's words and audio are posted on the merge request. Wrong words fail the pipeline, even when every unit test passes.
  4. Release and monitor. A person merges. Main deploys the demos to Cloud Run and NVDA listens to the live site.

What happened on the rating bug (MUI #42562)

  • Try 1 passed all the automated tests. NVDA in Chrome said "2 of 23". The pipeline failed it; a person re-assigned the issue and the flow got NVDA's words.
  • Try 2 still said "2 of 6" in Chrome.
  • Try 3: "2 of 5" in Chrome and Firefox. Merged into our GitLab copy (MR !2), deployed, and heard again on the live site.

Four more open reports

The flow fixed four more Material UI reports that are still open upstream, each heard right by NVDA in both browsers and merged into our copy: Chip read as "clickable" with no click handler (#49511), Popover content not announced as a dialog (#37613), IconButton losing focus while loading (#49477), and SpeedDial actions unreachable by arrow keys (#30875). After the last merge NVDA checked the live site: 5 fixes, 2 browsers, 10 of 10. Nothing was sent to the Material UI team.

GitLab's own agent made the same mistake

Given the same #42562 report, GitLab Duo's Developer flow wrote the same kind of first fix (aria-setsize on each star). All 50 Rating unit tests pass and its pipeline was green, because it wrote no listening test, so its NVDA job had nothing to play ("No journeys for this MR"). Played separately with our journey, NVDA in Chrome says "2 of 23".

A fair test, reported honestly

We put 12 old, already-solved Material UI screen reader bugs back into the code, hid the maintainers' tests for them, and gave each bug to the Unheard flow and to GitLab's Developer flow in two identical copies. The case list and its SHA-256 hash were published before any run. Both solved 10 of 12. NVDA heard all 24 fixes working. So Unheard does not write better code; its value is the check itself, which applies to any agent's fix and refuses the ones that only look right.

How we built it

  • GitLab Duo Agent Platform: a custom flow in the AI Catalog (journey agent and fixer agent), triggered by assign and mention.
  • GitLab CI/CD across nine stages: plan (nightly schedule, issues with NVDA's words), create (flow, draft MR), verify (NVDA proof in two browsers, MUI unit tests, JUnit in the MR), secure (scope check from the base commit, SAST, secret detection, keyless cloud identity), package (demo bundles, VM agent), release (human merge, Cloud Run deploy), configure (VM, autologon watchdog and agent from code), monitor (NVDA on the live site, nightly sweep), govern (transcripts and audio on every MR, a pre-registered answer key).
  • Google Cloud: Windows Server 2022 spot VMs running NVDA 2026.2, Cloud Run, Cloud Build, Artifact Registry, Cloud Storage, Secret Manager, Workload Identity Federation.
  • Guidepup and Playwright drive NVDA and the browsers.

Challenges

NVDA needs a real interactive Windows desktop, so the VM keeps an autologon session alive and retries when Windows will not focus the browser. Chrome and Firefox expose the same markup differently, which is exactly why the rating fix passed in Firefox and failed in Chrome. The audio in the video and on merge requests is NVDA's own speech log re-voiced with a Windows voice; NVDA's words are its log, unchanged.

What's next

Run Unheard on any GitLab project with a component demo, and add more screen readers (JAWS, VoiceOver).

Built With

Share this project:

Updates

Submission history