Logo

Inspiration

We've all had this call: "No, the other blue button. Top left. No, your other left." The person we're helping can't see what we mean, and we can't point. Screen sharing doesn't help, because the senior is the one who has to share their screen, and that is the hard part. Remote-control tools take over their computer, which is exactly what scammers ask them to install. We wanted something in between: a way to point at their screen without touching it.

What it does

Ollie is a Chrome extension that runs on the senior's computer. During a Google Meet call, it listens to the family member. When they say "click Compose," the rest of the senior's page dims and the Compose button stays lit, with a short instruction beside it.

The family member can speak or type the request, and they don't need to install anything.
If the request is ambiguous, Ollie shows numbered choices instead of guessing.
An optional "Hey Jarvis" wake word keeps all audio on the computer until the phrase is heard.
Ollie only points. It never clicks, types or submits anything.

How we built it

Ollie is a Manifest V3 Chrome extension with four parts:

Part	What it does
Offscreen document	Plays the Meet audio back, detects speech, runs the on-device wake word (ONNX Runtime Web), and cuts speech into short clips
Service worker	Runs the session, queues and deduplicates requests, calls Gemini, and decides what to point at
Content script	Scans the page's buttons and links, keeps snapshots of them, checks the target, and draws the spotlight
Popup	The listening switch, a text box for typed requests, settings and diagnostics

For each request, the content script sends Gemini a compact table of the page's visible controls: [id, role, name, region, visible]. Gemini gets that table plus the audio clip and returns structured JSON with the transcript, the intent and the id of the matching element. We check every answer against a strict schema. When Gemini isn't confident, a deterministic local matcher takes over. Before drawing anything, the content script makes sure the element still exists, is visible and still has the same name.

The extension has 124 unit tests and 51 integration tests. The integration tests load the real extension in Chrome and check, among other things, that it never clicks anything.

Challenges we ran into

Capturing Meet audio. Chrome's tabCapture mutes the tab it captures, so the senior would stop hearing the call. We play the audio back from an offscreen document. MV3 service workers can also be shut down at any time, so after every restart the session has to work out whether capture is still running.

Latency and overlapping clips. Sending one clip per sentence makes Ollie slow, but cutting clips at a fixed length splits sentences in half. We send a clip about 0.7 s after speech ends, or every 4 s during long speech. Each clip holds up to 6 s of audio, so consecutive clips overlap:

$$ t_{\text{overlap}} = t_{\text{clip}} - t_{\text{interval}} = 6\,\text{s} - 4\,\text{s} = 2\,\text{s} $$

That way a sentence that crosses a boundary is always heard whole. The overlap means the same request can arrive twice, so we drop repeats of the same target within 10 s. We also discard results that arrive after Stop or after a newer highlight. The time from speech to spotlight is roughly

$$ T \approx t_{\text{end of speech}} + t_{\text{upload}} + t_{\text{Gemini}}, \qquad t_{\text{end of speech}} \approx 0.7\,\text{s} $$

Pointing at the right thing. Real pages are messy. Some buttons are only icons, and some pages re-render between the question and the answer. A request like "the settings button" can match several things. We fall back on accessible names, re-match by role and name when the page changes, and show numbered choices when a request is ambiguous. Page text is also untrusted input: a page could contain text written to trick the model. So we JSON-encode it in the prompt, and Ollie only accepts element ids that were in the list it sent.

Accomplishments that we're proud of

It works on real sites, including Google Meet, YouTube, Gmail and Google Docs.
It's safe by design. It never acts for the user and doesn't store any audio or transcripts. Passwords and field values never leave the page.
There's no backend. Typed commands work offline.

What we learned

Chrome MV3 internals. We learned how service workers, offscreen documents, tabCapture and content scripts fit together, and how much each one is limited by Chrome's security model.

Designing for seniors. The best UI is almost no UI: one bright target, a few words beside it, and a big way to make it go away. We also learned that not acting on the user's behalf is a feature. It keeps the senior in control, and it's what makes Ollie safe to trust.

What's next for Ollie

Any call, any page. Support calling apps beyond Google Meet, plus pages Ollie can't read today, such as cross-origin iframes.
Fully local. Speech recognition and matching that run entirely on the user's computer.
Chrome Web Store release, with a custom wake word and support for more languages.

Installation

We are currently in the process of making Ollie available in Chrome extension store which may take a few days. In the meanwhile, you can install the extension directly from source in your own computer. A video is provided below showing how to do this: https://www.youtube.com/watch?v=AgimrjWJyQI

Built by Yasaman Parhizkar, Zahra Saraylou, Shiva Soleimany and Milad Moshayedi.