Ai Voice Browser

Inspiration

The inspiration for this project stems from the growing need for accessible web browsing solutions. Many individuals with physical disabilities or motor impairments face challenges when navigating the web, often relying on assistive technologies that are either limited or cumbersome. We wanted to leverage AI to create a more intuitive, seamless, and empowering experience for users who need voice-driven navigation to interact with the web. By combining speech recognition with Google's powerful AI capabilities, we can transform how people interact with websites, making browsing more inclusive and accessible for millions.

What it does

This extension uses AI-powered voice recognition to allow users to control their browser hands-free. With simple voice commands, users can navigate pages, scroll, play videos, search the page, click links, and refresh content. The extension uses Google's built-in Prompt AI API to understand and process the user's speech commands, then translates those commands into browser actions. The result is an intelligent voice assistant that responds to natural language and adapts to the user’s needs, enabling a more accessible browsing experience for everyone, especially those with mobility challenges.

How we built it

We built the extension using TypeScript and React for the frontend, providing a smooth and responsive interface. The core of the functionality leverages Google's Prompt AI API, which processes the speech commands and translates them into browser actions like navigating, scrolling, clicking links, performing searches and refreshing pages. For speech recognition, we used Web Speech API to capture and process the user’s voice input. The AI listens for commands, interprets them contextually, and then sends instructions to the browser to perform the corresponding actions.

Challenges we ran into

The main reoccurring challenge was integrating the Prompt AI API to ensure that voice commands were understood accurately and translated into the correct browser actions. Also, simulating events to search and insert text is not always reliable across various sites.

Accomplishments that we're proud of

We’re particularly proud of how well the AI adapts to different user commands and contexts. We’re also proud of the accessibility focus—we've made sure the extension works without the need for additional hardware or complex setups, allowing users with various disabilities to use it right out of the box. The voice commands are intuitive, making the entire browsing experience feel more natural and fluid.

What we learned

We gained valuable insights into the challenges of integrating AI-driven speech recognition with browser navigation. We also learned a lot about how AI can be harnessed to enhance accessibility, and how important it is to build inclusive solutions that consider a wide range of users and needs. We also explored how to make browser extensions as responsive and efficient as possible, ensuring that they don’t interfere with the user's browsing experience.

What's next for the Voice Navigator

Looking ahead, we plan to continue improving the accuracy and responsiveness of the voice commands. We also plan to extend the list of supported commands as we continue to iterate. The goal is to make the web more accessible, intuitive, and user-friendly for everyone, regardless of ability.

Built With

• chrome
• javascript
• prompt-api
• react
• typescript