AEyes

Inspiration

Recent breakthroughs in Artificial Intelligence, particularly object detection and image classification, and the proliferation of mobile technology at cheaper and cheaper prices open lots of exciting possibilities to improve the daily lives of disabled people. Inspired by the rapid advancements in computer vision and the omnipresence of mobile phones in our daily lives, we were wondering how the small yet incredibly powerful device we use every day could improve the lives of disabled people. In our brainstorming process, the range of possible applications we could think of turned out to be extremely vast, but because of the extremely tight time constraints of a 36-hour hackathon we chose to limit our efforts to developing an application that could help blind people navigate around the world.

What it does

AEyes uses the mobile camera to continuously take images, compress them and send them to a neural network for object detection that runs in the cloud. The neural network returns the objects detected in the given image, and sends them back to the phone. The phone then uses a text-to-speech system to read out the objects detected. Another feature is fEyend: hold the touchscreen of your phone and say what you are looking for. Your phone will then tell you to keep moving around the room until it finds the object you’re looking for.

How we built it

In order to be able to run our application on both iOS and Android without having to write code in both Swift and Kotlin, we wrote our application in React Native. While we originally planned to use Azure, we ended up going with AWS Rekognition to detect the objects because its performance on our test data was significantly better. After we finished our idea, every member of our team worked on a different part of the app: using the AWS APIs to upload the images and extract information from them, continuously taking pictures and compressing them in real time and building the user interface.

Challenges we ran into

The time we had to develop our app was very limited. Some of the technologies we tried did not have sufficient accuracy or speed to be used for our application. For example, the latency on the Azure neural text-to-speech APIs was too high, so we ended up using the native alternative that runs instantly but doesn’t sound as natural. Another problem was that the Azure SDKs are not optimized for React Native, so we ran into lots of problems and eventually decided to go with simple POST requests using a standard library for REST API calls.

Accomplishments that we're proud of

We were able to develop our first functioning prototype within 8 hours. This was a big relief because many parts of the development process were plagued by errors and we often had to adapt our plans to limitations we were not aware of before and did not have enough time to address.

What we learned

It is best to initially limit your idea to a narrow domain after the brainstorming session and ship an MVP. Additional features can be added later.

What's next for AEyes

AEyes does not have to be the only feature of the app: It would be very well possible for it to be part of a more generalized application that leverages the power of mobile and cloud computing to help people with all kinds of disabilities. After starting the application, the user could select the disability they cope with with, which would make available a specific set of tools tailored exactly to their needs. Some of these tools could include: location tracking and detection of unusual patterns in daily routines of people with Alzheimer’s disease, verification of faces or voices (e.g. whether a person is a relative or someone trying to scam you out of your savings) for people with dementia and a voice recognition service running in the background that calls/texts a list of relatives if it recognizes sentences asking for help. The societal impact of such an application would be unparalleled and the scope of future improvements is nearly limitless taking into account the continuously evolving capabilities of AI, increasing data transfer rates provided by 5G and thus further the shift in computational limits away from the small handheld device we all possess to gargantuan data centers. This is why we would be incredibly grateful if you support our idea. Thank you for your attention.