Handrail

Firestore Database showing user preference saved
Extension showing the Cloud Run - Web Socket connection in the extension
Cloud Run Log Shows Gemini 3.7 Flash Model Use

Inspiration

While working in IT industry and being part of Quality Engineering team, I observe how real customers move through a shopping journey, and the pattern that stays with me is not the one in the dashboards. It is the customers who abandon a basket not because they changed their mind but because the site defeated them. Older customers, mostly. They get through search and browse fine. They fall over at checkout, where the form fields multiply and the error messages are vague and the layout shifts under them.

You can fix a lot of that with better design, and we do. What you cannot fix with design is a customer who cannot see the screen at all.

The thing that turned it from a work observation into a weekend project was watching a close friend's grandmother try to order her groceries. She was using a screen reader. It read her 340 links before it reached a banana. She got as far as the delivery form, could not tell which field the error was attached to, and gave up and phoned my friend to do it for her. That is a person who wanted to spend money on a site that wanted her money, and the software stood in between them.

She has done that every week for years. My friend is her checkout.

What it does

Handrail is a voice agent that carries out tasks on real websites. You hold a button, say "find bananas and add the second one to my basket", and it does it, narrating as it goes, in the browser you are already signed in to.

The part I care about is what it does at checkout. It fills in your email and your delivery address without ever receiving either one.

Here is the actual mechanism. The extension reads the page into an accessibility tree on your machine. Before anything leaves, personal values are replaced with tokens. What reaches Google Cloud looks like this:

[e15] textbox "Delivery address" value=<vault:address.home>
[e22] button "Place order - $84.20"   [GATED: irreversible]

The agent plans against the tokens. It replies with type_text(ref="e15", value="<vault:address.home>"). The extension resolves that token from chrome.storage.local and types the real string into the real tab. The value never crosses the network, and the task still completes.

Prices, product names and stock levels are not redacted, because the agent needs them and they are not personal. That distinction is the whole design.

Three other things matter:

Your voice does not leave either. Transcription runs on device, through Chrome's built-in Prompt API, which is Gemini Nano. Only the resulting text reaches Cloud Run, exactly as if you had typed it.

The Glass Box. The side panel shows the exact outbound payload, live, and two badges say where speech and redaction are actually happening. Someone who cannot see the screen is being asked to trust an agent completely. This is how they, or whoever helps them, check that the trust is warranted.

It stops before it spends your money. Any control classified as irreversible cannot be clicked. The agent has to read the consequence back with the exact amount and get a yes, spoken or clicked. That classification happens server side, from the snapshot, not in the prompt, so a confused or manipulated plan still cannot place an order.

And it talks while it works. A sighted user infers progress from a spinner. There is no audible equivalent, so each action announces itself: "Opening the page", "Selecting Add to basket", "Filling in your email for email address". Without that, a twenty second journey is indistinguishable from a crash, and the user interrupts, and the journey is lost.

How I built it

Two halves and a WebSocket.

The cloud brain is a Google ADK 2.8 agent on Cloud Run running gemini-3.7-flash. It has no browser. It declares browser tools, but every call is relayed down the socket and executed in the user's own Chrome, and the coroutine simply awaits the result. From the agent's point of view it is an ordinary async function.

The client is a Manifest V3 extension. A content script walks the DOM into [ref] role "name" value="" lines and holds a map from each ref to the live element. The side panel owns the socket, the redactor and the vault. It runs in a document rather than a service worker on purpose, because a service worker gets evicted and I did not want a journey dying halfway through a checkout.

No Playwright, no MCP, no automation library. A click is chrome.tabs.sendMessage to the content script, which calls .click() on the element. That works because it is the user's real tab: already signed in, already past any bot check, nothing to detect. It is also why the agent needs no credentials to be useful.

Redaction runs in three passes, all local. First the vault match, so anything you have saved becomes a fillable token. Then deterministic rules for what regex is genuinely good at: emails, phone numbers, and card numbers verified with Luhn. Then Gemini Nano, with a JSON schema constraint, for what rules miss. A name in a greeting. An order reference in a confirmation banner.

That third pass is why the client is a Chrome extension rather than a desktop app. The built-in Prompt API exists only in Chrome itself; a bundled Chromium ships no on-device model, so redaction would have had to happen in the cloud, which is the exact thing this project exists to avoid. A constraint that decides an architecture is worth stating plainly.

Firestore holds preferences. Journeys are designed to store the vault references the agent used, never the values, so a journey stays replayable without anything personal having been recorded.

Stack: Google ADK 2.8, Gemini 3.7 Flash on Vertex, Gemini Nano via Chrome's Prompt API, Cloud Run, Firestore, Chrome Manifest V3.

Challenges we ran into

A live object cannot live in ADK session state. I stored the client bridge in session.state. InMemorySessionService round-trips state through a serialisable copy, so it came back as None and every browser tool would have failed at runtime with "no browser is connected", after deploy, with no obvious cause. Caught it in a five line script before the first deploy. There is now a test that asserts the failure mode still exists, so nobody moves it back.

A tool that raises kills the whole turn. ADK wraps the exception, the root node dies, and the user hears nothing at all. On a page that rewrites itself while you are reading it, which is most large sites, a routine stale element reference did exactly that. Tools now return failures as results, with a fresh snapshot attached so the model can pick a new reference and carry on. Refs carry a descriptor of role and accessible name too, so a replaced node can be found again.

String.replace passes the match offset where you expect a capture group. My card detector read the second callback argument as a captured string, so the Luhn check was being handed a number and calling .replace on it. Any page containing a card-like number would have thrown inside redaction. Code review would never have caught that. A test with a real checkout page in it caught it in ten seconds.

The echo suppression ate the user's actual instructions. With the microphone left open, the speakers feed back into it and the agent transcribes itself, so I compared each transcript against what had just been said. Then Handrail asked "would you like to add the item to the trolley?", the user answered "add the item to the trolley", and every word matched. I had built a product that ends every turn by offering concrete options, then written a guard that scored the user repeating one of those options as maximally suspicious. The fix was to stop judging anything that did not begin while Handrail was speaking, and my tests had all passed because I had invented user replies that happened not to reuse the agent's vocabulary.

Chrome's speechSynthesis lies about whether it is speaking. It cuts off utterances past roughly fifteen seconds without firing onend, and leaves speaking stuck true afterwards. Every gate in the panel keys off that answer, so when it sticks, progress narration goes silent and the microphone will not open for a confirmation. One stuck flag, and the two things a blind user most needs both stop. I diagnosed this correctly, half-fixed it, and it came back twice before I stopped trusting the API entirely and gave every utterance its own watchdog.

The speech model was not where I needed it. gemini-3.5-transcribe-live is not published on Vertex at the global endpoint, and the Gemini API route needs a billed key. Chasing that is what made me look properly at something I should have seen on day one, which is in the next section.

Accomplishments that we're proud of

The thing I am pleased with is not a feature, it is a test:

ok - nothing personal survives redaction, and everything useful does

It stands a browser up in jsdom, runs the real content script against a real checkout page, runs the real redactor over the output, and asserts both halves at once. The name, email, address, card, phone and order reference are all absent from the payload. The product names, prices, stock levels and button labels are all still present.

Either half alone is easy. You can redact everything and have a useless agent, or redact nothing and have a useful one. Both together, provable, in one assertion, is the project.

What we learned

Privacy is an architecture, not a feature. I could have added a redaction step to a cloud browser agent and called it private. It would not have been. The value would still have been in the cloud, one prompt injection away. The only version that holds is the one where the value is never there, and that is a decision you make on day one about where the browser runs, or you do not get to make it at all.

I had a hole in my own argument and a billing failure found it. For most of the build, speech went to a cloud transcriber. When that broke and I went looking for a local option, I had to admit the thing I should have seen at the start: a voice is biometric, and I was streaming it to a server in a project whose entire claim is that nothing personal leaves your machine. "We only send the audio, not the page" is not a distinction anyone should be asked to accept. Transcription is on device now.

Constraints that look like limitations are usually the product. "The agent cannot see your address" sounds like an apology. It is the whole pitch. Same with the confirmation gate: an agent that refuses to complete the last step is a better agent for this user, not a worse one.

Write the test that embarrasses you. The worst bugs in this build were all found by test files in the first ten minutes they existed, and none of them were visible by reading the code I had just written.

What's next for Handrail

Replayable journeys. The history is designed to store references rather than values, which means "do my usual Thursday shop" can resolve locally at replay time and run mostly unattended.

Real user testing, and this is the honest gap. Everything here is my judgement about what a visually impaired user needs read aloud. My judgement is not evidence.