Looking at pictures is very nostalgic for the majority of people, but not for all. Blind people are inherently unable to experience such emotions through sight. Nevertheless they have memories that can be triggered by auditory inputs. Our mission is to tell blind people their stories, thorugh their pictures and our voice.
What it does
The mobile app allows people to select images from their own gallery and hear a detailed description of the picture. Moreover it can tell what you can see in a given direction (left, right, center), look for a specific thing (glass, lion, table) and (not yet implemented) share your favourite pictures on social media! Everything completely voice controlled.
How we built it
We integrated Alexa Echo with the Clarifai API and the Dropbox storage, everything cosily connected in an Android app. Here the detailed pipeline: The user opens our app, slides the gallery pictures, touches one (ANY!), then asks Alexa Echos 'What's the picture about?'. The picture is sent to a dropbox folder where Alexa picks it up and sends to the Clarifai API. Clarifai tags the picture, the tags are rendered by our AWS Lambda Functions and the final sentence is returned to the user by Echo.
Challenges we ran into
What we just described in few words seems pretty smooth, but the challenges are countless. Integrating so many different services together, developing a mobile app and designing a super easy user interface is amazing. Being able to do all of this in 36 hours is not hard, is incredibly fun. The truth is that we almost though we could not make it, but, the team was to dedicated to fail. The end result just made us forget the countless hours of effort and frustation.
Accomplishments that we're proud of
What we learned
Skill we had: Java, Python, Friendship Skill we have now: Android mobile development, RESTful services, AWS Alexa, Friendship,
What's next for My Eyes
Well, let us tell you a story, your story. Close your eyes. Alexa, narrate Bob's life.