You know that someone is clearly in front of you. Who is it? You know that both Emily and Jane have shortish hair, a roundish face, but the world is much too blurry for you to correctly discern what is happening You wish there were a way for you to know instantly who was in front of you without always having to ask. But instead, you're left in constant darkness with what is going on around you.
Because of that need, our team developed a cool and interesting way to connect the visually impaired with the world around them using Android phone technology and Facebook.
What it does
Spot-A-Friend allows visually impaired members of the community to better see the world around them. The process is simple. You open up the Spot-a-friend App and place the phone into the VR headset. From there, the view the user sees will be that of the Android camera view. The person can then look around and, when they feel that there are people in the area whom they can't identify, they can press a button on the headset and potentially utilize a voice command. "Spot a friend".
From there, the friend is identified and their general location
How we built it
We built it by interfacing the Android and a computer back end system. The android app takes cares of the image by capturing it via button command. The entire system will be attached to a VR headset and mounted in the eye region. It interfaces via python flask back end that utilizes socket technology. It re-purposes Facebook's tagging algorithm to be used as a facial recognition. Then, the back end returns information on what the algorithm was able to identify.
Challenges we ran into
Not enough food and not enough sleep! Also trying to do facial detection while streaming the camera was difficult. Syncing up speech recognition and text to speech functions also posed a great challenge!
Accomplishments that we're proud of
Communication between the back end and the android app was difficult. Also, maintaining both systems in syncs and having the back end bypass csrf on Facebook's domain and internal endpoints was a huge success!
What we learned
Teamwork and how to balance work and play :) !
What's next for Spot-a-Friend
Use Microsoft's Azure services to incorporate sentiment analysis so that the user can understand the context and emotion of the people around them. Additionally we would like to iron out the kinks with our speech recognition feature, and allow more commands from it. Perhaps, we can even reduce the form factor of the entire device so that people can wear it around with ease.