AccessPilot — Making the Web Accessible, One Action at a Time

Inspiration

Imagine trying to apply for a scholarship, fill out an important form, or navigate a government website when your screen reader announces nothing but "button, button, image."

For millions of people with visual impairments, this is a real barrier. Despite advances in web accessibility, many websites still contain unlabeled buttons, confusing forms, inaccessible date pickers, and visual-only error messages. Even experienced screen-reader users can struggle to complete everyday online tasks independently.

Meanwhile, AI agents are becoming increasingly capable of navigating websites and performing actions on behalf of users. But we noticed a critical gap: most AI browser agents focus on getting the task done, not on keeping the user informed, empowered, and in control.

That inspired us to build AccessPilot, a voice-first, accessibility-focused AI browser assistant that doesn't just navigate websites for users — it explains what it's doing, asks permission before important actions, and makes complex web interactions more accessible.

Our guiding principle was simple:

AI should empower people to navigate the web independently, not take away their control.

What It Does

AccessPilot helps users complete complex website tasks using natural-language instructions, voice commands, or keyboard interactions.

Instead of manually navigating every button, dropdown, and form field, a user can simply say:

"Help me complete this scholarship application."

AccessPilot then:

1. Understands the webpage: Analyzes the page and generates an accessible summary of its structure, including forms, buttons, and important sections.
2. Plans the task: Uses AI to understand the user's goal and identify the steps needed to complete it.
3. Navigates automatically: Interacts with the website through a real Chromium browser, clicking buttons, typing into fields, scrolling, and navigating between pages.
4. Explains every step: Provides natural-language progress updates, accessible announcements, and browser screenshots so users know what's happening.
5. Asks before important actions: Pauses before submitting forms or performing other consequential actions, allowing users to review details and explicitly approve or decline.
6. Protects sensitive information: Hands control back to the user for passwords, payment details, and CAPTCHA challenges.
7. Lets users stop anytime: Supports an immediate Stop button, keyboard shortcut, and voice command.

AccessPilot also detects whether someone uses their own screen reader. If they do, it avoids unnecessary duplicate speech and communicates through accessible live announcements.

The result is an AI assistant designed around user independence, transparency, and informed consent.

How We Built It

We designed AccessPilot as a modular, multi-component AI system combining language understanding, visual computer use, browser automation, and an accessible frontend.

1. Gemma 4 — The Brain

We integrated Gemma 4 through the Gemini API to handle high-level reasoning and communication.

Gemma helps AccessPilot:

• Understand the user's natural-language goal.
• Interpret webpage context and identify important interface elements.
• Break complex tasks into manageable steps.
• Generate accessible page summaries.
• Narrate progress in clear, understandable language.

This allows users to focus on what they want to accomplish instead of figuring out how each website works.

2. Twinly — The Eyes and Action Planner

We integrated Twinly's computer-use API to interpret browser screenshots and propose the next action.

Twinly receives the current screenshot, the user's goal, relevant webpage elements, and previous action history. It then determines whether to click, type, scroll, press a key, wait, or finish the task.

We also built an action-normalization layer to translate Twinly's responses into browser-compatible commands.

One interesting engineering challenge was handling coordinate inconsistencies between normalized model coordinates and actual browser pixels. We developed coordinate-resolution and element-matching logic to improve the reliability of interactions.

3. Playwright — The Hands

We built a browser automation engine using Playwright, TypeScript, and Chromium.

This engine handles:

• Creating isolated browser sessions.
• Navigating permitted websites.
• Capturing screenshots and extracting webpage context.
• Executing AI-generated browser actions.
• Verifying action outcomes.
• Detecting stale observations and invalid interactions.
• Stopping running actions and cleaning up browser sessions.

Rather than allowing an AI model to execute arbitrary browser operations directly, our browser controller validates actions and applies safety checks before execution.

4. Next.js and React — The Accessible Interface

We developed the frontend using Next.js, React, TypeScript, and Tailwind CSS, with accessibility as a core design requirement.

The interface includes voice input, spoken narration, keyboard shortcuts, task progress tracking, browser previews, approval dialogs, and screen-reader-compatible status announcements.

We used accessible interface patterns such as ARIA live regions, logical focus management, visible keyboard focus indicators, and explicit text descriptions instead of relying only on colors or visual cues.

5. The Orchestrator — Connecting Everything

We created a backend orchestrator that connects Gemma, Twinly, Playwright, and the frontend.

Our architecture follows this workflow:

User Goal → Gemma Planning → Twinly Action Proposal → Playwright Execution → Screenshot and Verification → Accessible Feedback

The cycle repeats until the task is completed, stopped, or requires user intervention.

We use HTTP APIs and Server-Sent Events (SSE) to communicate task progress, screenshots, approvals, and status updates to the frontend.

We also implemented an explicit approval mechanism: before a consequential action, the system pauses, presents the details, and waits for the user's decision.

Challenges We Faced

Making AI Automation Safe and Predictable

A major challenge was balancing autonomy with user control.

AI agents can perform actions quickly, but a mistaken click on "Submit," "Delete," or "Pay" can have real consequences.

We addressed this by implementing multiple safety layers, including action validation, browser-side detection of consequential interactions, explicit approval IDs, and safeguards against duplicate or stale actions.

Our goal wasn't just to make the AI capable of acting — it was to make its actions accountable.

Translating AI Predictions Into Reliable Browser Actions

Twinly interprets screenshots visually, while Playwright interacts with a browser using viewport coordinates and DOM information.

These representations don't always align perfectly. We encountered coordinate differences that could cause an action to miss its intended target.

To address this, we implemented coordinate normalization, element-label matching, and safeguards against executing actions based on outdated screenshots.

Building for Accessibility From the Beginning

Accessibility isn't achieved simply by adding a microphone button.

We had to consider how users would understand progress, review approvals, recover from errors, and interrupt tasks without relying on visual information.

We built keyboard navigation, screen-reader announcements, voice commands, and accessible approval flows into the core experience rather than treating them as optional additions.

Integrating Multiple AI and Automation Components

Gemma, Twinly, Playwright, and the frontend each have different responsibilities, data formats, and execution requirements.

We defined shared TypeScript contracts and built a modular orchestrator to keep these components independent while allowing them to work together.

This made it possible to develop and test modules separately, use mock AI responses during development, and connect the real services as they became available.

Handling Real-World Reliability

Browser automation involves asynchronous navigation, slow-loading pages, unexpected redirects, changing page layouts, and network failures.

We added timeouts, cancellation handling, bounded retries, URL restrictions, action verification, and automated tests to make the system more reliable.

Accomplishments We're Proud Of

Our biggest accomplishment is building an integrated AI browser assistant with accessibility and user consent at the center of its architecture.

We're especially proud of:

• Combining Gemma 4, Twinly, and Playwright into a modular browser-agent workflow.
• Building an accessibility-first interface that supports voice, keyboard, and screen-reader interaction.
• Implementing explicit approval gates before consequential browser actions.
• Creating a real Chromium automation engine with screenshot-based observations and action verification.
• Developing a scholarship-application demonstration that showcases form completion, approval, and confirmation.
• Writing automated tests for browser execution, safety checks, approval handling, stopping, and orchestration.

Rather than relying entirely on a prerecorded sequence, we built the underlying components needed for AI-driven browser interaction, alongside reproducible mock demonstrations for reliability.

What We Learned

Building AccessPilot taught us that creating a useful AI agent requires much more than connecting a language model to a browser.

We learned how to coordinate multiple AI components, translate visual predictions into executable actions, manage asynchronous browser sessions, and design reliable human-in-the-loop workflows.

More importantly, we learned that accessibility and AI safety reinforce each other.

When an AI assistant explains its actions, provides meaningful feedback, and requests approval, those features don't just benefit users with disabilities. They make the system more understandable and trustworthy for everyone.

We also gained practical experience with multimodal AI, computer-use APIs, Playwright automation, accessible React development, TypeScript contracts, and end-to-end testing.

What's Next for AccessPilot

We envision AccessPilot growing into a more capable accessibility companion for everyday web tasks.

Our next steps include expanding support for different websites, improving interaction with highly dynamic interfaces, adding more languages, and testing directly with people who use screen readers such as NVDA, JAWS, and VoiceOver.

We also want to improve personalized task assistance, error recovery, and the ability to resume interrupted workflows while maintaining strict user approval requirements.

Ultimately, our vision is to make the web easier to navigate for people who are too often excluded by inaccessible design.

The future of AI shouldn't just be about what agents can do. It should be about who they empower.

That's why we built AccessPilot.

Built With

• chromium
• gemini-api
• gemma-4
• javascript
• next.js
• node.js
• playwright
• react
• server-sent
• tailwind-css
• twinly
• typescript
• vitest
• web-speech-api
• zod