Inspiration
The inspiration behind this is utilizing the advances of technology to serve as a third eye for those who are visually impaired. We wanted to create a technology that would be in hand that is just as useful, if not more useful, than just a white cane or any other sort of visual aid.
What it does
The app captures and identifies any objects that are on-screen using Tensorflow. When the user simply taps on the screen, using depth perception, the phone is able to tell the user what the object is and how far away it is. It can also tell you the relative orientation from the phone (e.g. "slightly to the left", "slightly to the right", or "directly in front"). This app also is completely on device, so those who do not have internet access or those who can't afford internet are free to use this without such burden. This app also works on any camera, where no need for a lidar or dual stereo.
How I built it
It was built in Android Studio with two Tensorflow models, one of which serves to find depth in images and the other to detect objects in images.
Challenges I ran into
Some of the challenges that we ran into were working with Android Studio in general. Debugging was a bit tedious, and the wait times to rebuild Gradle was often frustrating. There was also some initial concern processing depth perception, live, long to process, so we had to change up a bit of how we wanted the app to work.
Accomplishments that I'm proud of
We are proud of having built an app that would help aid those that are visually impaired, and to help potentially improve our community. We are also proud that there are other accessibility capabilities, such as not needing a specific camera, or not requiring internet access.
What I learned
We learned to use services, like Google Cloud, in being able to train and recognize objects. We also learned how to develop apps for Android a bit better. Most importantly, through the thought process and designing this app, I think that this taught us to try to think from a perspective much different from ours.
What's next for annotate-the-world
Better depth perception, and better accuracy overall.
Log in or sign up for Devpost to join the conversation.