Access Pilot

AccessPilot Workspace — AI-powered browser automation with live previews, task progress, and conversational assistance.
Personalized Autofill — Save your details locally for faster, safer form completion.
Keyboard Accessibility — Quick shortcuts for voice control, stopping tasks, and navigating hands-free.
Voice-First Experience — Navigate websites using natural language, with screen-reader support and step-by-step guidance.

Inspiration

Picture applying for a scholarship with a screen reader. The form has unlabeled buttons, a date picker that won't work with the keyboard, and a Submit button hidden below the fold. Every field becomes a small battle, and one wrong press could send an application you haven't finished.

Or imagine navigating that same form with limited mobility, or on a day when a long, complicated website feels overwhelming.

For millions of people, the web still presents barriers that shouldn't exist.

Today's AI assistants can explain how to complete a form, and newer browser agents can perform actions automatically. But we saw a missing piece: What if an AI could actually do the work with you, while making sure you always understand and control what's happening?

That's why we built AccessPilot — a voice-first AI browser assistant that makes the web easier to navigate, one action at a time.

Our philosophy is simple:

Tell me what you see. Explain the page and narrate every step.
Help me get things done. Navigate complicated websites instead of just giving instructions.
Respect my privacy. Hand sensitive information back to me.
Keep me in control. Never submit, pay, send, or delete without my explicit approval.

Accessibility shouldn't mean giving up independence. And automation shouldn't mean giving up control.

What it does

AccessPilot lets users interact with websites through natural language, voice commands, or keyboard input.

Give it a website and tell it what you need.

Ask questions about any page

Instead of struggling to navigate an unfamiliar website, users can ask:

"How many fields are there to fill?"
"Summarize this page."
"What information do I need to log in?"
"What is the application deadline?"

AccessPilot analyzes the page's readable content, including content beyond the visible viewport, and uses Gemma 4 to answer in plain language.

In our testing, it correctly identified 40 fields on a real page.

Users can understand a website without having to interact with every element manually.

Complete real browser tasks

Say:

"Help me complete this scholarship application."

AccessPilot understands the goal, creates a plan, and uses AI-driven browser automation to click, type, scroll, and interact with the website.

While it works, users receive accessible progress updates, spoken narration, and a browser preview.

It doesn't simply say "Done" when an AI model thinks the task is finished. It checks the resulting page for evidence of success.

Your information, your control

Users can save their name, email, education details, and other relevant information under Settings → Your details.

These details are stored locally in the user's browser and can be supplied to the agent for a task.

If AccessPilot needs information that hasn't been provided, it pauses and asks instead of inventing an answer.

The latest interface also supports answering these follow-up questions directly inside the application.

Safety comes first

AccessPilot is designed to keep users in control of consequential actions.

Explicit approval: Before submitting forms, sending information, making payments, or deleting something, AccessPilot pauses and requests confirmation.
Sensitive-field handoff: Passwords, payment-card information, and CAPTCHAs are handed back to the user.
Immediate Stop: Users can interrupt a task using the interface, a keyboard shortcut, or a voice command.
Accessible feedback: Screen-reader users receive ARIA live announcements, while other users can enable spoken narration.
No guessing: Ambiguous approval responses never count as permission.

The goal isn't just autonomous browsing. It's trustworthy, accessible automation with a human in control.

How we built it

AccessPilot combines three core technologies — Gemma 4, Twinly Computer Use, and Playwright — through an accessible Next.js application and a custom orchestration backend.

Gemma 4 — The brain

We use Gemma 4 (gemma-4-26b-a4b-it) through the Gemini API for language understanding, task planning, and accessible communication.

Gemma powers:

Natural-language task planning.
Page understanding and accessible summaries.
Questions and answers about webpage content.
Identifying missing user information.
Step-by-step progress narration.
Explaining task results in understandable language.

We built structured prompts, schema validation with Zod, retry handling, and safeguards that treat website content as untrusted input.

We also optimized the model's reasoning settings to reduce planning latency without sacrificing the accuracy observed in our tests.

Twinly Computer Use API — The eyes

Twinly's Computer Use API (POST /v1/computer/next-action) determines the next browser interaction from a live screenshot.

For each step, Twinly receives the current screenshot, the user's goal, previous action history, and hints about labeled page elements.

It proposes an action such as clicking, typing, scrolling, or pressing a key.

We built a TypeScript adapter to validate and normalize those actions before sending them to our browser controller.

Twinly also provides signals for consequential actions and sensitive values, which we connect to AccessPilot's approval and handoff workflows.

Playwright and Chrome — The hands

We built a custom browser automation engine using Playwright, TypeScript, and Google Chrome/Chromium.

The browser engine:

Launches isolated browser sessions.
Navigates websites within configured restrictions.
Captures screenshots and extracts page context.
Executes validated browser actions.
Detects stale screenshots and invalid clicks.
Blocks unapproved consequential actions.
Verifies action outcomes.
Supports cancellation and cleanup.

We also implemented coordinate correction and element matching to make screenshot-driven interactions more reliable.

Next.js — The accessible experience

Our frontend uses Next.js, React, TypeScript, and Tailwind CSS.

It includes:

Push-to-talk voice input and spoken narration.
Keyboard shortcuts and screen-reader live regions.
Accessible approval and handoff dialogs.
Real-time task progress and browser previews.
Light and dark themes.
Saved user details and speech preferences.
A redesigned, responsive workspace.

The interface targets WCAG 2.2 AA accessibility practices, with automated accessibility checks covering multiple interface states.

Orchestrator and real-time communication

A custom backend coordinates the entire workflow:

User Goal → Gemma Planning → Twinly Action → Playwright Execution → Verification → Accessible Feedback

The orchestrator manages task state, approval requests, missing-information prompts, failures, and cancellation.

We use HTTP APIs and Server-Sent Events (SSE) to stream task updates to the frontend.

Five team members developed the system in parallel using shared TypeScript contracts, automated testing, and GitHub pull requests.

Our repository now includes 36 merged pull requests, approximately 1,100 automated tests reported by the team, and CI checks covering linting, type checking, unit tests, and browser-based end-to-end tests.

Challenges we ran into

1. Twinly's coordinates didn't always match the browser

Twinly sometimes returned coordinates using its documented 0–1000 grid and sometimes returned pixel-based coordinates.

That caused clicks to miss their intended targets.

We developed a coordinate-resolution system that interprets ambiguous coordinates both ways, compares them against labeled webpage controls, and selects the most plausible target.

We also added element-matching safeguards for consequential clicks.

2. An AI saying "Done" doesn't mean the task succeeded

Early versions could report completion even when a browser interaction had failed.

We learned that an AI-generated success message isn't enough.

We improved action verification, failure handling, and completion checks so that task results are grounded in what actually happened on the webpage.

The website's confirmation matters more than the model's confidence.

3. Websites change while the agent is working

Dynamic layouts, loading states, scrolling, and page transitions can invalidate previously planned actions.

Our browser controller rejects clicks based on stale observations.

We added page-settling logic and refreshed observations between actions to reduce failures caused by changing interfaces.

4. Understanding the entire page was harder than expected

Initially, answering questions from only visible controls caused AccessPilot to miss information farther down a page.

It could also incorrectly infer whether fields were required.

We expanded page-context extraction to include readable page text beyond the viewport and refined Gemma's instructions to avoid inventing information the page doesn't provide.

This improved answers to questions about forms, requirements, and page content.

5. AI latency affected the user experience

Early planning calls could take close to a minute.

We measured the latency and experimented with Gemma's reasoning settings. Using faster inference for appropriate tasks reduced planning to a few seconds in our tests.

We also improved retry handling for transient model API failures.

Our measured scholarship workflow improved from approximately 90 seconds to 60 seconds, although actual timing varies with the models and network.

6. Making autonomy safe without making it frustrating

We needed to distinguish ordinary navigation from actions with real consequences.

An unnecessary approval prompt interrupts the user, but missing an important approval could be dangerous.

We combined model-provided action metadata with browser-side safety checks, improved approval wording, and refined how dropdowns and date controls are handled.

The result is an agent that can work independently while stopping when the user's decision genuinely matters.

Accomplishments we're proud of

Built a working AI browser-agent architecture integrating Gemma 4, Twinly, and Playwright.
Verified a live Twinly-driven scholarship workflow completing a five-field application in approximately 12 steps with one submission approval.
Added whole-page question answering, including a verified 40-field counting example.
Implemented personal-detail autofill with prompts for missing information.
Built accessibility into the core interface, including narration, screen-reader support, keyboard navigation, and voice commands.
Created safety mechanisms for consequential actions, sensitive information, stale observations, and task cancellation.
Improved real-world reliability through coordinate correction, page-settling logic, retries, and result verification.
Built an open-source project with an MIT license, contributor documentation, automated tests, and a CI pipeline.
Prepared a containerized deployment configuration for Railway, with support for deploying the integrated application from the main branch.

What we learned

The biggest lesson was that accessibility isn't a feature you add at the end — it's a design principle that changes how the entire system works.

It influenced how we structured our AI agents, how we communicate progress, how we handle errors, and when we return control to the user.

We also learned that building a reliable AI agent involves much more than connecting an LLM to a browser.

Models can misunderstand coordinates. Websites change unexpectedly. API calls fail. A successful-looking action might not have worked.

Every one of those challenges pushed us to build stronger validation, clearer feedback, and more reliable verification.

Most importantly, we learned that AI autonomy and human control don't have to compete.

The best assistant isn't the one that acts without asking. It's the one that knows when to act, when to explain, and when to let the user decide.

What's next

We want to take AccessPilot beyond our hackathon prototype and make it useful across a wider range of real-world websites.

Our next goals include:

Browser extension: Bring AccessPilot into the tab users already have open.
More natural conversations: Expand spoken responses to clarification questions and follow-up prompts.
Multilingual accessibility: Support narration and interaction in more languages.
More reliable automation: Improve handling of dynamic websites, complex forms, and unexpected errors.
Real accessibility testing: Work directly with people who use NVDA, JAWS, VoiceOver, and other assistive technologies.
Persistent workflows: Explore safely resuming tasks after interruptions and server restarts.

Our long-term vision is simple:

A web where people don't have to fight the interface to accomplish what matters to them.

Prize categories we're targeting

Best Use of the Twinly API

Twinly isn't a decorative integration in AccessPilot. It's the core computer-use decision engine.

Twinly's POST /v1/computer/next-action endpoint chooses the next browser action from live screenshots.
Its consequential-action signals feed into our human-approval workflow.
Sensitive-value signals trigger user handoffs instead of automated secret entry.
Our integration includes coordinate correction, labeled-element hints, bounded requests, and explicit error handling.
We tested a live end-to-end scholarship workflow with approximately 12 Twinly actions and one final approval.

Our innovation is making visual AI computer use accessible, explainable, and accountable.

Best Use of Gemma (MLH)

We use Gemma 4 (gemma-4-26b-a4b-it) through the Gemini API as AccessPilot's understanding and communication layer.

Gemma:

Understands user goals and creates task plans.
Distinguishes questions about a page from requests to perform actions.
Generates accessible page descriptions.
Answers questions using extracted webpage context.
Identifies missing information.
Narrates browser actions and results.

We also implemented structured-output validation, resilience against transient API failures, untrusted-page-content handling, and latency optimizations.

Gemma makes a visual, action-oriented browser agent understandable to people who cannot rely on visual feedback.

Best Open Source Project (MLH)

AccessPilot is publicly available under the MIT license.

GitHub: https://github.com/varshith-k/AccessPilot

The repository includes:

A modular TypeScript architecture and documented integration contracts.
Developer documentation for the AI, browser, and orchestrator modules.
A contributing guide, code of conduct, and GitHub collaboration templates.
Automated unit, integration, accessibility, and browser tests.
GitHub Actions CI.
36 merged pull requests across a five-person team.

We built AccessPilot to be understandable, reproducible, and extensible by other developers.

Main Awards and Vobile Prize

AccessPilot addresses a meaningful problem: many people still encounter unnecessary barriers when using everyday websites.

Our project combines:

Real-world impact: Making complex web tasks easier for blind, low-vision, motor-impaired, and other users facing accessibility barriers.
Technical innovation: Combining Gemma 4 reasoning, Twinly computer use, and Playwright execution.
A working prototype: Real browser automation, page understanding, form completion, and approval-based submission.
Human-centered safety: Explicit consent, sensitive-field handoffs, transparent narration, and immediate cancellation.
Open-source engineering: A modular implementation with automated tests and documented architecture.

AccessPilot isn't about replacing the user. It's about removing the barriers between the user and what they want to accomplish.