Web Whisper

Inspiration

Browsing the web is an integral part of modern life, but it often falls short when it comes to multitasking. Whether reading an article, following a recipe, or diving into a novel, the need to constantly interact with the browser can interrupt the flow, especially during hands-on activities like cooking or exercising. This challenge inspired us to create Web Whisper, a tool that makes hands-free browsing seamless, enabling users to interact with web pages using only their voice. Our goal is to enhance productivity and accessibility for all.

What it does

Web Whisper is a Chrome extension that seamlessly integrates voice-command functionality into your browsing experience. With just a microphone and your voice, users can navigate web pages, interact with forms, and even perform complex tasks like searching, scrolling, or clicking buttons. It recognizes context-sensitive commands and adapts to various website structures, offering a smooth and intuitive way to surf the web without touching a mouse or keyboard. Perfect for multitaskers, accessibility users, or anyone who wants to embrace the future of hands-free browsing!

How we built it

Web Whisper is built as a Chrome extension, utilizing the Chrome Extensions Platform for seamless integration with the browser. The front-end is crafted with HTML, CSS, and JavaScript, ensuring a clean and responsive interface. For speech recognition, we relied on the window.SpeechRecognition API, enabling the extension to capture and process voice commands in real-time.

For more straightforward commands, such as navigation or interaction with elements on a webpage, we implemented custom logic that maps voice input to specific actions. For more complex or less direct commands, we used the Google-built Prompt API, which allows us to parse and interpret more nuanced instructions.

Additionally, to improve command matching and ensure flexibility, we incorporated Fuse.js, a fuzzy search library that helps us match voice commands to the corresponding actions with a high level of accuracy. We also used Git and GitHub for version control, ensuring that our code was well-managed and collaborative throughout the development process.

By combining these technologies, we created an intuitive hands-free browsing experience that adapts to various user needs.

Challenges we ran into

The hackathon journey was challenging, as it was our first proper one, and we faced multiple obstacles. We found out about the hackathon late, which left us with limited time to implement, test, and refine our features. Both of us struggled to fully commit due to ongoing exams, further limiting the time we could dedicate to the project.

Much of the technology we used, including the Chrome Extensions Platform and speech recognition APIs, was new to us, requiring extra time for learning and troubleshooting. One team member also faced issues setting up the AI, creating a divide between development and testing, which delayed integration. Additionally, being in different time zones made communication and coordination challenging. Despite these obstacles, the experience was invaluable for growth and learning.

Accomplishments that we're proud of

We're proud of completing as much as we did with the limited time and manpower. Moreover, we're really proud of our growth while working on this project.

What we learned

This hackathon was a significant learning experience in both technical and non-technical aspects. On the technical side, we learned how to work with the Chrome Extensions Platform, integrate speech recognition APIs, and utilize Fuse.js for fuzzy search, all of which were new to us. We also gained hands-on experience with prompt engineering with the Google Prompt API and how to handle voice commands of varying complexity. Beyond technology, we learned valuable lessons in time management, especially when faced with tight deadlines and the need to balance multiple responsibilities. Working remotely also taught us how to communicate effectively across time zones and manage a project with limited synchronous collaboration. Overall, we gained a deeper understanding of what it takes to successfully complete a hackathon project from start to finish.

What's next for Web Whisper

There're a lot more features we're planning to add to the extension. We also need to sort out some bugs and make the entire experience more fluid. We want to add more actions that can be performed, including navigating between tags an searching, as well as providing more feedback to the users.

Built With

• css
• fuse.js
• html
• javascript
• prompt-api