Spotlight

Inspiration

As tech-savvy CS majors, we are often bombarded by our relatives on tech-related topics, like how to renew a driver's license or how to schedule an appointment with their primary doctor.

What it does

Spotlight is a voice-guided web navigation assistant that helps users browse websites using voice commands and visual guidance. Spotlight highlights exactly where to click, making web navigation intuitive and accessible for everyone, especially older users and those with cognitive or visual challenges.

How we built it

We built a Google extension that harnesses the power of FishAudio and Claude. We first use FishAudio's Speech to Text to interpret a user's verbal request. Then, we put this into Claude alongside an image of the website to determine the correct answer as well as the positioning needed for the visuals. We then use FishAudio's Text to Speech to deliver the verbal answer to the user.

Challenges we ran into

Some challenges we ran into include:

computer vision scanning - being able to accurately find the correct position for visuals
voice recognition (WIP)
designing the user interface to be intuitive for our target audience

Accomplishments that we're proud of

We're happy to have a working demo by the end of the hackathon, especially since we had to fly in on Friday night (and so couldn't work on the project for a while).

What we learned

How to build an extension in Chrome, as well as how to call APIs for it
Knowledge of better methods for computer vision scanning
Things really do break towards the end
More detailed communication is very important for team collaboration

What's next for Spotlight

Voice recognition
Slight modification of UI to look more streamlined
Adding more assistive tools (ex. ability to choose color theme or high contrast for color blind users)
Fun features like customizable cursors!