Inspiration
Most websites are built for one kind of user. We kept working around pages instead of using them: zooming until the layout broke, squinting at gray-on-white text, opening YouTube for one video and losing an hour to Shorts. Every site has its own settings, if it has any. We wanted to just say what we needed and have the page change.
What it does
Cappy is a Chrome extension. Open the bar or start talking, say "make this easier to read" or "hide Shorts on weekdays 9 to 5", or point at something and say "make this bigger", and Cappy edits the page you're on. It checks the result against the page's own accessibility stats and rolls back anything that made things worse. Your saved changes come back on your next visit without another AI call. If you keep zooming or missing small buttons, it offers a fix, and it only remembers things about you after you say yes.
How we built it
Everything is TypeScript. The Chrome extension handles the bar, voice, camera gestures and the actual page edits. A small local server runs the agent and keeps the API keys off the browser, and it works with ChatGPT, Claude, OpenAI or Gemini. The model never writes JavaScript. It writes CSS that we scan first, uses a short list of allowed page edits, or calls building blocks we wrote ourselves, like dark mode, a colorblind filter and a form fixer. Page text is treated as untrusted, so a page that tries to give Cappy instructions gets blocked. Playwright tests drive real Chrome against a fake model.
Challenges we ran into
Latency was the big one. Every request waits on a model, and even saved changes took almost 3 seconds to come back on a busy page, because we re-checked each one from scratch. Skipping that check for changes we'd already verified got 12 saved changes down to 0.19 seconds. New sites used to sit blurred for up to 8 seconds; now it's 1.5.
Getting rid of Shorts was harder than it sounds. YouTube puts them in shelves that look different on every page, and at one point our filter hid the entire search results instead of just the Shorts. Now it removes a whole shelf by its heading, and it tells you if anything it claimed to hide is still on screen.
We went back and forth on the UI for a long time before settling on a Spotlight-style bar with settings built in. Voice was its own fight: deciding when someone has finished talking without cutting them off mid-sentence.
Accomplishments that we're proud of
We built something we'll actually use. Not a demo we'll forget next week, but a tool we want running in our own browsers after the hackathon.
What we learned
A lot about voice. Detecting the end of a sentence from silence is fiddly, because rooms are never quiet and a microphone's automatic gain turns up background noise the moment you stop talking. We also learned why turn-by-turn voice feels slow. Transcribing, then thinking, then speaking leaves long gaps, and newer voice models can listen and talk at the same time.
What's next for Cappy
A real voice conversation with GPT-Live, where Cappy talks back while it changes the page. It's built and we're testing it now. After that: asking before a rule runs on a new site for the first time, and trying Cappy on more real sites, like government forms.
