Blip | Devpost

What it does

Blip lets you control Chrome with your voice. Say what you want to do, and Blip interacts with the browser and talks you through the result.

We built Blip with blind and low-vision users in mind, bringing browser navigation, page reading, and everyday interactions into one conversational interface.

Control your browser: Open websites, search the web, switch tabs, open new tabs, go back and forward, and scroll using spoken commands.
Understand what’s on the page: Ask “Where am I?”, hear page content, list links, request a summary, or ask questions about what you’re reading.
Read and respond: Read supported messages aloud, summarize them, and draft or revise a response.
Interact with websites: Dictate text into supported fields and hear values read back.
Confirm consequential actions: Sending a message or closing a tab requires a separate confirmation.
Choose how you interact: Hold a keyboard shortcut to speak, type a command, or use the screen-reader output mode.

How we built it

Browser integration: A Chrome Manifest V3 extension reads page structure and carries out browser actions. It combines accessibility-tree information with DOM inspection to identify content, links, fields, and controls.

Local coordination: A Node.js server connects the extension and voice interface over WebSockets. Shared TypeScript contracts keep commands, page updates, and responses consistent across the system.

AI assistance: Common browser commands follow predictable code paths. Gemini, accessed through the Antigravity CLI, handles open-ended requests, page questions, summaries, and drafting. Model output is validated before it can affect the browser.

Speech: Moonshine handles on-device speech recognition, and Kokoro generates spoken responses in the browser. Push-to-talk provides a clear listening window, while captions show what Blip heard.

Action safeguards: Confirmation rules live in application code. Blip checks the current page and action target before acting, filters sensitive information from model context, and treats webpage instructions as untrusted content.

Testing: We built local test pages and automated checks for browser commands, changing page state, inaccessible fields, confirmation flows, and malicious page instructions.

Challenges we ran into

Keeping up with changing pages: Tabs switch, pages navigate, and controls disappear. We had to check fresh page information before acting so an old command would not affect the wrong target.
Coordinating speech: Listening, transcription, playback, and keyboard shortcuts all need to work together. Clear listening cues and careful timing became central to the experience.
Understanding inconsistent websites: Missing labels and unusual page structures make reliable interaction difficult. We combined accessibility information with DOM heuristics and asked for clarification when a target was ambiguous.
Turning requests into safe actions: Understanding what someone said is only part of the problem. We also needed to validate the target, check the action, and request confirmation when necessary.

Accomplishments that we're proud of

Connecting spoken commands to real browser navigation and page interaction.
Expanding Blip from an accessible form assistant into broader voice control for Chrome.
Bringing on-device speech recognition and spoken responses into one interface.
Supporting page reading, questions, and drafting alongside everyday browser commands.
Building confirmation checks into the execution path from the beginning.

What we learned

A voice interface needs to explain where the user is, what happened, and what comes next.
Common commands benefit from predictable code, while AI helps interpret language and explain content.
Completing an action does not prove the intended result happened. Reading the page again matters.
Accessibility shapes the entire interaction, including response length, listening cues, confirmations, and error recovery.

What's next

Test with blind and low-vision users and improve the experience around their feedback.
Expand support for complex websites and custom controls.
Improve response speed, speech accuracy, and recovery from misunderstood commands.
Simplify installation and extension setup.
Expand beyond our current Windows setup.