The realization of the recent explosion in computer vision accuracy paired with an intense drive to build something that could actually solve a real-life problem led us to the decision to build an app that would help the visually impaired see. Inspired by Be My Eyes, a free app that connects blind people with sighted volunteers, we quickly decided that this was the most meaningful project we could work on. The ability to see is a central part of all of our everyday lives. When we arrived at MHacks, we decided we wanted to work on a project that would help those who suffer from blindness find their way around.

What it does

Conceptually, White Cane is as simple as a white cane. Technically, however, it is not so trivial. Using image recognition, object detection, and multi-label classification, we created an app that informs blind people of their surroundings. By tapping the screen, a user is informed via audio output from their device of the general situation they're in, as well as specific objects and their relative positions.

How we built it

  • Swift front-end that interacts with Cloud Functions on Google Cloud Platform
  • Google Cloud Vision API for multi-object detection using object localization
  • Firebase MLKit to analyze text in images
  • Firebase Cloud Storage to store image files

Challenges we ran into

  • Running Python code from a Node.js server
  • Unwanted image deformation on iOS
  • Determining the distance to objects in images lacking a reference object

Accomplishments that we're proud of

  • Creatively utilizing a myriad of APIs including the Cloud Vision API
  • Constructing meaningful descriptions from image tags
  • Designing locational summaries from object detection data

What we learned

  • Flask
  • What it feels like to stay up two nights straight
  • The power of git diff
  • What it’s like to work on a project that's actually meaningful
  • Learned about making APIs and using Google APIs

What's next for White Cane

  • Conversational interface
  • Distance from objects (virtual white stick)
  • Launching on the App Store
Share this project: