2D WebMCP - helping Blind users working with AI in 2D UIs

Blind people already use AI to use 2D UIs, but they have to jump through hoops to verify the output. This project makes verification easier using WebMCP, so blind users get AI's full benefit.

Comment 1

Inspiration

Screen reader tech has sophisticated ways of navigating web content, but reading complex UI such as diagrams, maps, charts, boards, slides, etc can quickly break down. With enough engineering investment, these applications can be exposed accessibly, often through alternative presentations (for example a visually hidden table of data), but even then they require a lot of navigation and working memory to understand the whole picture.

AI can make this easier by interpreting and operating on the user's behalf, but there’s still a gap in the verification step. Spot-checking a result can mean laboriously retracing a complex UI, asking a sighted person, or trusting the agent without an independent way to verify its work.

The same people who get the most out of AI cannot always get the proof they need to trust it.

This is true even for serial content like an IDE: a study of 16 programmers who use screen readers backs this up, showing that the most time-consuming part of their workflow was verifying agent output. Participants had a hard time tracking the scope and location of changes, verifying results across multiple views and determining if the agent had completed every requested task.

What it does

The application does two important things. The more novel of the two is a pattern for managing focus in the application. It exposes a WebMCP tool to deep link to a part of the UI for evidence of an action. effectively a shortcut to proof of a change.

The second aspect of the project is a response contract + schema that includes fields for changes in the application’s state, so that agents and screen reader users can verify the change. The list of state changes can support other evidence the agent gathers from the UI state (via screenshotting or the accessibility API diffs), but most importantly, they can be inspected by a screen reader user to get confidence.

Example

  1. User makes a request in the agent to add a shape to a canvas
  2. Agent invokes WebMCP tool to add the shape
  3. The tools responds with a success payload with the updates as JSON (coordinates, type, content, errors, etc), along with a deep link to the closest focusable DOM node for inspection
  4. Agent interprets the payload, describes the change to the user.
  5. User asks the agent to focus the change in the browser via a tool call.
  6. The user's screen reader announces the change entry.

Impact

An assistive technology user can hand off a complex task without giving up the ability to fully understand the result. Verification is fast and reliable and doesn’t replay the same complexity the agent abstracted away. At scale, this can materially improve user confidence, remove barriers, and unlock creativity.

How I built it

I built a workflow editor with a 2D editable canvas that exposes tools to agents (and proximately to screen readers) via WebMCP. These include evaluation, editing, and undo tools. Most notably, it also includes a tool to focus an element. This is the real innovation in this project, and has broader applications. I also honed in on a balanced set of tools and schemas for generic 2D editing context. The output schemas are especially important in this demo, since they are what give the user improved confidence in the changes.

What I learned

I tested two variants of the tool design through an end to end workflow, editing, reconciling, revising, and focusing objects. One variant had conventional tool descriptions and minimal response schemas, and one with a more comprehensive and detailed output. The detailed output is qualitatively important for trust and verifiability, at the cost of roughly 45% additional input tokens per trial (very little change in output tokens). I'd hoped to also see better task performance from the agent with more detailed outputs as a consolation for the additional token spend, but they performed the same (nearly 100% success rate with gpt-5-mini for both). Dataset in here.

It bears mentioning that my baseline may have been generous in its definition of a "basic" output schema. Applications may respond to MCP tools (native or web) with simple “success” response object with insufficient structured content for efficient chained actions. WebMCP has even weaker constraints, with callbacks returning Promise<any>, and in fact Chrome's docs show returning a simple success string. I'm curious to see the role that shared schemas, conventions, and best practices have on efficient agent interactions with WebMCP over time.

What's next

I'm documenting the conventions around how to provide structured evidence from tool use of a change in 2D canvases, and also as a model for domain-specific schemas broadly; these schemas will benefit screen reader users and agents as WebMCP proliferates and evolves to support output schemas and more sophisticated docs and real world examples. I'm hopeful that some of this research and evidence can help companies working on 2D-based UI improve their agentic readiness with WebMCP.

Research

Blind screen-reader users have described the cognitive burden of working with two-dimensional artboards even when object details and coordinates are available:

“It’s too much information and not enough at the same time.”

Schaadhardt, Hiniker, and Wobbrock, Understanding Blind Screen-Reader Users’ Experiences of Digital Artboards · Evidence discussion

Accessible tables are familiar and necessary, but they can still make overview and comparison difficult. One participant facing 393 rows said:

“I can’t really get a snapshot.”

Zong et al., Rich Screen Reader Experiences for Accessible Data Visualization · Evidence discussion

Blind users already test and cross-check AI rather than treating it as automatically reliable. One participant advised:

“Just play with it. Don’t rely on them.”

Alharbi et al., Misfitting With AI: How Blind People Verify and Contest AI Errors · Evidence discussion

Built With

Share this project:

Updates