AccessPilot

Verified AI assistance for VoiceOver users, with explicit approval before sensitive input and fresh proof after every browser action.

Comment

Inspiration

Blind and low-vision people often reach forms that are visually understandable but poorly exposed to assistive technology. A missing label or ambiguous control can stop an otherwise routine task. We wanted to explore a narrower question: can AI help interpret broken semantics without becoming an unbounded browser agent?

What it does

AccessPilot is a development-only macOS companion for VoiceOver users. The user describes one bounded task in plain English while staying in the current Chrome tab. AccessPilot separates observed page facts from GPT-5.6 semantic inference, presents the planned action in an accessible command workspace, and requires a one-time confirmation immediately before every sensitive L2 field input.

After each approved action, AccessPilot takes a fresh page snapshot, re-resolves the target, and verifies a fresh postcondition. A dispatched event is never treated as proof of success. The demonstrated Travel Lab task completes three confirmations, four browser actions, four fresh verifications, six results, and returns focus to the real Chrome result.

How we built it

The system combines an Electron macOS companion, a Chrome Connector extension, a Swift native-messaging relay, closed TypeScript protocols, and an isolated Codex CLI profile. GPT-5.6 receives a redacted and aliased semantic snapshot, never a browser handle. It can propose bounded semantic annotations and a plan, but only local deterministic policy can create an authorized executable step.

Renderer code has no Node, shell, credential, connector, or raw action authority. Passwords, OTPs, payment data, identity data, CAPTCHA, and all L3/L4 flows are filtered before model or action paths. AccessPilot never inserts ARIA, overlays, labels, styles, or focusability into the page as an accessibility repair.

Built with Codex and GPT-5.6

Codex was used throughout Build Week to turn the requirements into typed boundaries, adversarial tests, accessibility-focused UI, packaging checks, and reproducible demo assets. GPT-5.6 powers the product semantic interpretation step through a closed runtime schema. The current actual-context semantic window completed 20 of 20 runs without invented references, tool events, or product/provider failures.

Challenges

The hardest part was preserving user authority across four trust boundaries: renderer, desktop main process, native messaging, and the browser page. We also had to keep model inference visibly distinct from DOM evidence, make confirmation reject-first for VoiceOver, verify postconditions from fresh snapshots, and keep sensitive values outside model input and logs.

Accomplishments

  • A coherent command-first product experience using real development-package screen recordings.
  • Three value-bound L2 confirmations and four independently verified browser actions in the demonstrated task.
  • 1,068 TypeScript and Vitest checks plus 55 Swift tests, 1,123 total.
  • A deterministic English demo with burned-in captions and no simulated product behavior.
  • Fail-closed schemas and policy boundaries that make L3/L4 execution unrepresentable.

What we learned

The useful role for a model is interpretation, not authority. Accessibility assistance becomes more trustworthy when the product shows what the page actually exposed, what AI inferred, what the user approved, and what was freshly verified as four different things.

What is next

The next gates are a complete human VoiceOver auditory and keyboard matrix, clean-account onboarding, and broader research with blind macOS users. A signed and notarized release is intentionally outside this Build Week submission.

Built With

Share this project:

Updates