During one of Sarthak's visits to the Samarthanam School of the Blind and Disabled in India, he realized the challenges many people face as a result of poor vision. So, for this hackathon, he suggested doing a project that could utilize advances in computer vision and machine learning to improve quality of life for those who are visually impaired.

What it does

Aids visually impaired users in locating desired objects, such as chairs, keys, laptops, etc, by using voice commands for queries, and object detection to recognize if image is in direction of phone camera. As the user moves the phone around their surrounding, the moment GlassEye detects the object it gives back Haptic Feedback to the user in the form of vibrations, so the user can go and pick it up.

How we built it

  • Tensorflow Object Detection API
  • Preprocessing for finding annotated images
  • Images converted to TFRecords
  • Trained with R-CNN on Tensorflow
  • Model was saved as tensorflow .pb file after being trained
  • .pb was converted to a frozen tflite for Android
  • Android app records with camera until it finds the object from voice input

Challenges we ran into

Preprocessing Images and Training R-CNN with Tensorflow Object Detection was difficult because of dependency conflicts. Converting pb file to tflite was also challenging.

Accomplishments that we're proud of

We are proud that our app is complete and readily implementable, with full accessibility to the visually impaired users through voice input and haptic feedback.

What we learned

How to train new classes using Tensorflow's Object Detection API, for identifying locations of objects rather than regular CNNs we built earlier. This is the first Android App any of us built, and it was a big learning experience.

What's next for GlassEye

  • Train to recognize more objects through user upload.
  • Create iOS version.
  • Integrate with voice services like Alexa, Google Assistant, Siri.
  • Publish app on Google Play Store to make publicly accessible, extending its implications beyond the hackathon.
Share this project: