Amazon Alexa and other voice AIs have the potential to dramatically improve the lives of those living with disability. We believe that Alexa has the potential to be used as a voice interface, allowing visually impaired citizens to augment their senses using a voice interface.
What it does
It allows a user to ask Alexa to either read text in an image or describe the image to them.
How we built it
The Alexa Skill calls an AWS Lambda endpoint that runs using Flask Ask, and determines whether the user wants text in before them read out loud via Google Cloud Vision or whether they want the image before them described using Microsoft's Project Oxford. The Lambda function sends a request to the user's computer through ngrok and takes a picture using the computer's webcam and uploads it to an Amazon S3 bucket.
Challenges we ran into
Originally we wanted to use a GoPro that could be strapped to the user's head but unfortunately the public GoPro APIs are very limited. We ended up switching to using a webcam in a VM. We also had plenty of trouble setting up the Lambda functions using Zappa, but we were able to pull through in the end. Originally we were planning on doing the Optical Character Recognition (OCR) on our own and we invested a fair bit of time into it before deciding to use a 3rd party API. We also wanted to change the Alexa invocation to something less confusing but it broke things.
Accomplishments that we're proud of
Actually getting it to work. Seriously, this is one of the most complete hackathon projects any of us have had. Getting all the different parts of the backend to work together, to culminate in a nice Alexa Voice User Interface (VUI).
What we learned
We learned a lot about Alexa skills! Most of us came into this weekend with no prior knowledge of Alexa and now we know a whole lot more.
What's next for Eyelexa
Eyelexa is not that practical of a skill for certification on the Alexa web store because of its numerous moving parts. We're not sure what direction Amazon will move in the future with Alexa, but we believe Eyelexa has potential as a standalone device with a camera included for possible private use. We could potentially build a device using the Alexa Voice Service to enable Alexa on a device worn on one's person.