Inspiration
The realization of the recent explosion in computer vision accuracy paired with an intense drive to build something that could actually solve a real-life problem led us to the decision to build an app that would help the visually impaired see. Inspired by Be My Eyes, a free app that connects blind people with sighted volunteers, we quickly decided that this was the most meaningful project we could work on. The ability to see is a central part of all of our everyday lives. When we arrived at MHacks, we decided we wanted to work on a project that would help those who suffer from blindness find their way around.
What it does
Conceptually, White Cane is as simple as a white cane. Technically, however, it is not so trivial. Using image recognition, object detection, and multi-label classification, we created an app that informs blind people of their surroundings. By tapping the screen, a user is informed via audio output from their device of the general situation they're in, as well as specific objects and their relative positions.
How we built it
- Swift front-end that interacts with Cloud Functions on Google Cloud Platform
- Google Cloud Vision API for multi-object detection using object localization
- Firebase MLKit to analyze text in images
- Firebase Cloud Storage to store image files
Challenges we ran into
- Running Python code from a Node.js server
- Unwanted image deformation on iOS
- Determining the distance to objects in images lacking a reference object
Accomplishments that we're proud of
- Creatively utilizing a myriad of APIs including the Cloud Vision API
- Constructing meaningful descriptions from image tags
- Designing locational summaries from object detection data
What we learned
- Flask
- What it feels like to stay up two nights straight
- The power of
git diff - What it’s like to work on a project that's actually meaningful
- Learned about making APIs and using Google APIs
What's next for White Cane
- Conversational interface
- Distance from objects (virtual white stick)
- Launching on the App Store
Log in or sign up for Devpost to join the conversation.