VisionBuddy

Inspiration

We are inspired by the potential of powerful computer vision systems to _ make a difference _ in the life of individuals with visual impairments.

What it does

VisionBuddy is an application that helps individuals with visual impairments get a vocal description of the objects in their surroundings. When the application starts, the user is vocally instructed to press long on the screen to get a description of the environment. Since VisionBuddy is a fully immersive Android application, the user can press anywhere on the screen to activate the vocal description. The application can identify the categories found in the COCO dataset.

How we built it

VisionBuddy is based on TF Detect, which uses Tensorflow Object Detection API as published in Speed/accuracy trade-offs for modern convolutional object detectors to "localize and track objects (from 80 categories) in the camera preview in real-time".

Challenges we ran into

Figuring out the camera ratio to fit the whole screen in order to give an immersive experience was not straightforward. Integrating vocal instructions was challenging.

Accomplishments that we're proud of

Getting a tensorflow model to run on a mobile device!

What we learned

We expanded our knowledge in Android programming and tensorflow.

What's next for VisionBuddy

The next step is to add a Visual Question Answering agent to VisionBuddy. We are also interested in adding a localization feature that "remembers" the location of the objects of interests.