Reach

chrome extension
firestore for rag memory
demo electricity webapp
google cloud

Inspiration

My neighbour’s grandmother lost her sight, but she still wanted to shop online, pay bills, book tickets, and use the web just like anyone else.

The problem wasn't that she didn't know what she wanted to do. The problem was that many websites were designed for sighted users, leaving blind users unable to understand what buttons and icons mean or complete tasks independently.

That inspired us to build REACH — an AI browser agent that understands inaccessible web interfaces, lets a blind user simply say what they want, and automatically performs the task for them, while verifying the result and learning from their corrections.

Because losing sight shouldn't mean losing independence on the web.

What it does

Say "REACH" or hit Alt+R, and it wakes up, listens, and takes a goal in plain language — the same way you'd ask a person sitting next to you.

Say "open my electricity bill," and REACH observes the page, works out what matters, acts, looks again, and checks whether what you asked for actually happened. It doesn't plan five steps ahead and blindly run through them — the page changes after every action, so it works in a tight loop instead:

GOAL → OBSERVE → REASON → ACT → OBSERVE AGAIN → VERIFY

It reads every page two different ways, because each one fails differently.

A structural reading — the DOM, ARIA attributes, accessible names — fast and exact, but blind to anything a developer never labelled.
A visual reading — an actual screenshot, understood the way a sighted person would. It can spot a payment icon that the code never described, but it can't see hidden state.
In between, a smaller, faster model ranks the page's elements by relevance to the goal first, so the heavier reasoning only has to consider a handful of real candidates instead of dozens.

When those two readings disagree, nothing happens.

Structure → Cancel
Vision    → Pay Now
        ↓
     CONFLICT
        ↓
      BLOCK

"I found conflicting information about this button, so I won't activate it."

It's a conversation, not a set of one-off commands. Say "open my electricity bill," then later just "open it," and REACH knows what it means. Say "actually, show me payment history instead," and it drops the old goal without making you start over.

It remembers, and it changes because of it.

What it's learned about a specific website.
Corrections you've given it — "no, that icon is the payment button, not settings."
Your personal preferences — how much it reads back to you, whether you always want to confirm before payments.

All of that gets pulled back in on your next visit, which is why the fifth time you pay a bill through REACH is quicker and quieter than the first — it already knows the page, and it already knows you.

It checks its own work. A click landing isn't the same as a task finishing, so REACH inspects the page afterward for real evidence before claiming anything succeeded. And for anything consequential, it pauses and names the actual amount out loud before it acts — nothing moves until you say yes.

How we built it

REACH is three pieces, each doing exactly one job.

A Chrome extension lives in the browser and is the only part that ever touches the page — it reads structure, takes screenshots, and is the only thing that clicks, types or scrolls. It never reasons.
A backend on Google Cloud Run does all the actual thinking, using Google's Agent Development Kit (ADK) to coordinate a set of specialised agents.
Firestore sits underneath both as long-term memory — sessions, what's been learned about individual sites, corrections, preferences, and payment records.

We kept the extension deliberately dumb on purpose. It only observes and acts, which means REACH runs inside the user's own logged-in browser session and nothing has to leave it unnecessarily.

The models. Almost all reasoning runs on Gemini, reached through Vertex AI using Google Cloud's built-in credentials rather than an API key — there's genuinely no key anywhere in this project. Gemini interprets what the user means, understands the page structurally and visually, reconciles the two when they disagree, chooses the next action, and judges afterward whether it actually worked. Alongside it, Gemma, a smaller and much faster model, works purely as a relevance filter — ranking the page's elements against the current goal before the expensive reasoning ever runs, so a page with thirty interactive elements gets reduced to a handful of real candidates. Gemma never makes the final call and never touches the browser. If it fails or times out, the system just falls back to considering every candidate, as if Gemma had never run — an optional optimisation should never be able to take the whole pipeline down.

Why structure runs first. This isn't only about cost. For someone who can't see a loading spinner, every extra second of silence between speaking and REACH acting just feels broken. Running every agent — including the visual ones — on every single step would mean several seconds of dead air between actions, worse than the screen reader we're trying to help. So the structural agent runs alone whenever it's confident, and the visual agent only wakes up when labels are genuinely missing or confidence is low. Most steps end up costing a single model call.

The stack, end to end.

AI: Gemini and Gemma via Vertex AI, orchestrated with Google ADK
Backend: Google Cloud Run, FastAPI, Pydantic
Memory: Firestore
Browser: Chrome Extension (Manifest V3), Web Speech API for voice
Payments: Razorpay, test mode
Auth: Application Default Credentials throughout — no API keys anywhere

We built every reasoning step as its own agent in ADK, each returning a strictly typed, validated response instead of loose text we'd have to hope was parseable. We also deliberately turned off ADK's ability to let agents hand control to one another, because every routing decision — every skip, every hard stop — needed to live in plain, readable, testable code rather than inside a model's own judgment. When REACH skips the visual step, that's because a confidence number came back below a threshold, not because a model quietly decided it felt sure enough.

On data. REACH doesn't run on any external dataset. What it actually reasons over, every time, is:

The live DOM and accessibility tree of whatever page the user is on right now
A live screenshot of that exact page
Its own Firestore memory — what it's learned about that site, that user's past corrections, that user's preferences

To demonstrate this reliably rather than hoping a real website misbehaves on camera, we also built our own demo site — a fictional electricity portal, deliberately containing an icon-only payment button with no label, and a button where the visible text and the accessibility label directly contradict each other. Beyond that, we leaned on the WCAG and ARIA specifications as reference for how accessibility information is supposed to be exposed, which is what let us reason clearly about the ways real sites fail to expose it.

On payments. We integrated Razorpay in test mode so we could demonstrate a genuinely real order rather than faking one — but we deliberately skipped the normal checkout screen. A typical checkout ends in a card form and an OTP prompt, and that kind of modal, time-limited screen is its own accessibility problem, regardless of anything else REACH does well. Instead the order is created and settled entirely server-side, so the whole flow is hands-free, and the resulting receipt is what the verification step later reads as proof the payment actually went through.

Challenges we ran into

Getting a model to say something reasonable was never the hard part. Earning the right to actually let it click things on someone's behalf was.

Models invent things that don't exist. Ask a model to identify a button and it will sometimes describe a selector that isn't present anywhere on the real page. We ended up adding several separate layers of validation specifically for this, checking every proposed target against the real, live page before ever letting an action execute.

Structure and vision disagree more than we expected. We originally built the visual agent purely as a fallback for when structure came back blank. What we actually found was that its more valuable role was catching genuine disagreements — cases where the two readings pointed to completely different meanings. Those turned out to be exactly the situations most likely to hurt someone if the system guessed wrong, which is what pushed us toward a real hard stop instead of any kind of tie-breaking logic.

"Clicked" doesn't mean "done." A click can land perfectly and the intended outcome can still not have happened, especially with something like a payment that might still be processing. That's what forced verification to become its own separate step — running only after the page has genuinely had a chance to update, rather than bolted onto the end of the action itself.

Deciding what actually counts as risky took real thought. Scanning the user's stated goal for words like "pay" fails immediately — "pay my electricity bill" as a goal would trigger a confirmation gate on every related interaction, including harmless ones like just opening the bill to look at it. What actually made sense was checking the specific action and its target right before execution, not the broader goal behind it.

And then the ordinary grind: Firestore permissions, getting IAM roles right on Cloud Run, sorting out environment variable loading order, and — after a genuinely frustrating stretch of debugging — discovering that Gemma was only available on a global endpoint while all our Gemini calls were running in a specific regional one.

Accomplishments that we're proud of

We're genuinely proud that REACH works end to end, as a complete system rather than a scripted demo — a spoken goal turns into a real observation, a real decision, a real action, a real check, and a memory write that makes next time faster. We ran real end-to-end tests against the live models and a real payment order, not mocked responses, to make sure the whole pipeline actually holds together.

But if we had to pick one moment, it's not a case of REACH succeeding. It's the demo where it deliberately does nothing.

We built a button that visually reads Pay Now while its accessibility label says Cancel. Most browser automation we're aware of would just pick one and act. REACH recognises the two disagree, refuses to act at all, and explains why. There's no path anywhere in the system that lets an action execute once that disagreement is flagged.

We're also quietly proud of something smaller nobody watching a demo would notice on their own: the payment flow has no OTP screen anywhere in it. A judge might skim right past that. A blind user trying to get through a card verification screen unassisted would notice immediately.

Most browser-agent work we've seen gets measured on task completion rate. We think that's the wrong thing to optimise for here. When someone can't check your work themselves, being honest about uncertainty matters more than finishing.

What we learned

Building something that's allowed to act is a different problem than building something that just answers. A chatbot that says something wrong gives a bad response. An agent that acts wrong submits a form, sends a message, or moves someone's money — and that difference has to shape the architecture from day one, not get bolted on at the end.

Safety can't live inside a prompt. You can ask a model as nicely as you want to be careful, but you can't depend on it. Every decision in REACH that absolutely cannot fail — refusing on conflicting evidence, gating consequential actions, only ever reporting success when there's real evidence — runs as plain code, regardless of what the model said or how confident it sounded.

Verification matters as much as the action itself, maybe more. It's easy to make an agent click things. It's much harder to build one that can honestly tell you whether that click actually accomplished anything, and harder still to build one that says "I'm not sure" instead of guessing.

Accessibility isn't the same as readability. We started asking how to help blind users understand a website. We ended up asking how to help someone actually finish what they came to do, independently. A site can expose every string of text to a screen reader and still leave someone unable to get anything done on it.

And the honest one we're still sitting with: we built this for blind and visually impaired users, but not yet with them. Closing that gap — real testing, real feedback, real reshaping of the system — is the most important thing left to do.

What's next for Reach

REACH lives in a Chrome extension today. We think of that as a starting point, not the destination.

Onto more of the real web. Government portals, shopping platforms, utility providers, tested well beyond our own demo site, with nothing special required from the website itself — and getting more cautious as the consequences grow, not less.
Into the phone, natively. The right home for this isn't a browser extension. It's the operating system. Most web use happens on mobile, and mobile is where accessibility breaks hardest — smaller targets, gesture-driven interfaces, apps whose accessibility information is often thinner than a webpage's. Built into the phone, the same idea works across every app at once:

VOICE GOAL  →  SYSTEM-LEVEL AGENT
                      ↓
   ┌──────────────────┼──────────────────┐
   ↓                  ↓                  ↓
ACCESSIBILITY      SCREEN            APP ACTIONS
   TREE           UNDERSTANDING      (tap / type / scroll)
   └──────────────────┼──────────────────┘
                      ↓
              RECONCILE → ACT → VERIFY

Hold a button, say what you want in your own language, and the phone does it — then tells you honestly whether it worked. Not an app you open. Something that's just there, the way TalkBack and VoiceOver are just there.

Languages people actually speak. Kannada, Hindi and Tamil first — accessibility that only works in English quietly leaves out most of the country.
Memory that compounds across people, not just sessions. So what one person teaches REACH about a confusing, unlabelled site doesn't have to be relearned by the next person who hits the same wall.
Built with the people it's for. Real testing with blind and visually impaired users, letting what they say decide what actually gets built next.

The goal was never to describe the internet to someone who can't see it. It's to help them use it — on their own, without having to ask anyone.

We don't want blind users to keep adapting to an inaccessible web. We want the web to adapt to them.