One of our teammates has a friend who is visually impaired, and we thought this project would be a great way to demonstrate how image recognition and deep learning technology can be used as a "third eye" for the blind community, empowering them in their daily lives. Many members of the visually-impaired community are depressed and isolated as it's difficult for them to partake in the same daily rituals as non visually-impaired people. Through ThirdEye, we hope to empower the blind community with the technology to thrive in busy situations.
What it does
The smart glasses, equipped with ultrasonic sensors, camera, buzzers and a raspberry pi zero, analyze the user's surroundings. The glasses detect whom the user is talking to and objects around it through image recognition using a combination of personally developed image filtering in order to get the most important subjects in a photo in combination with the use of Google Vision and IBM Watson. We used a combination of three ultrasonic sensors running simultaneously on different threads in order to provide and process data in real time.
- By saying "Who is that?", the app will perform facial recognition on a person -We have also paired this facial recognition with the user's Facebook friends to find a match
- User can verify transactions through the Nessie API. If there's is an issue, the Capital One Service Line is called
- By saying "Count my money", the glasses will count how much money the person is looking at
- Using Nessie, we are able to take a picture of a check and deposit the amount in the user's bank account
- By saying "What is that?", the app will report with the best guess as to what the person is looking at
- Designed for urban living, the app will alert the user when nearing a stopped pedestrian crossing
How We built it
ThirdEye consists of 3 major components: a pair of smart glasses, a mobile app, and a node.js backend.
Challenges We ran into
None of us had used the RPi Zero, and given it's hardware limitations, it was difficult to configure and we had to optimize our scripts to consume less resources. We had to make multiple trips to stores such as Micro Center and CVS to obtain additional adapters and supplies to fit the RPi zero. We also had to run an older version of Node JS that was compatible with the RPi. On the hardware side, it was very challenging to hook up all the sensors and process the data from all the inputs simultaneously, all while fitting them along the sides of a regular pair of Google sunglasses. Furthermore, 3 out of 4 members of the team had very little experience with node.js, so we learned a great deal as we created the backend. The complexity of managing three components (deciding the roles of each, providing ways for all of the components to communicate, etc) proved to be a tricky yet surmountable task.
Accomplishments that we're proud of
For most of our teammates, this was our first time using node.js. We were able to successfully develop a deep learning model in order to provide confidence levels as to the object or person they see in front of them. Using various API's as well as many different optimization algorithms we were able to reach a high amount of accuracy. The "What is that?" voice command was very important to us, as it truly allowed a user to "see" what was around them.
What We learned
Our hack uses image recognition and deep learning algorthims which with the right application can make a lasting impact for the better for a visually-impaired individual. We learned how to interact with GPIO ports on a RPi to collect data, which, coming from arduino, is not an easy feat.
What's next for ThirdEye
Cleaning up the physical design (using a slimmer battery) as well as integrating all the components in a smaller form factor. Allowing the user to create their own voice commands catered to their personal lives.