Logo
Web App Layout
Phone App Layout

Inspiration

We have found that around 8 million people have a visual impairment disability that makes it more difficult for them to navigate through their lives. Thus, finding simple objects around the house becomes an obstacle. We decided to create an app called Whisper that uses objection detection to find the object by navigating the user in real time using auditory directions.

What it does

We decided to start off with our app as a web app as our starting point. The user would open Whisper, opening Whisper automatically opens the camera. Then, the user would tell Whisper to find an object. The user would hold their laptop camera and slowly move their laptop from side to side like a visual detector so that the app will be able to find the object. Once Whisper finds the object, Whisper will tell the user that they found the object and will give them real-time directions on how to get to the object.

How we built it

We crafted Whisper using Node.js to power our backend, ensuring robust performance and seamless data handling. To bring our vision to life, we utilized Figma for designing our web and mobile applications and logo, allowing us to prototype and refine our interfaces.

Challenges we ran into

One hurdle we ran into was integrating vector generative AI into our web application, which was essential for enhancing real-time navigation capabilities. Additionally, programming a Voice Assistant to work seamlessly within the app added more complexity to the program, requiring extra time and collaboration to ensure that both features functioned smoothly together. With most of the team being new to programming, initially, it was time-consuming to get up to speed with developing the back end.

Accomplishments that we're proud of

We are glad to have created a working demo that can seamlessly do object recognition despite being a novice team.

What we learned

We learned how to utilize Figma and React to accomplish our project's needs.

What's next for Whisper

We would add the Voice Assistant to the app. We would include different auditory elements and allow the user to customize them. We would want the user to be able to customize the voice of Whisper to make it more personable. We would also program Whisper to speak in different languages such as Spanish and Hindi to make it more accessible to those who do not speak English. We would also include a feature where the user would take a picture of an object that is not as easily recognizable by the app such as objects that have sentimental value. Once the user takes a picture, Whisper would save that picture in the memory bank. When the user wants to find that object, then Whisper would navigate the user to help find it. We would also want to expand our audience to people who have issues with memory retention and help them find objects.