We noticed there are no good applications that help blind people learn about their surroundings well. For example, Without the help of their families and friends, they even don't know what is in a photo they just shot.

What it does

This app provides a verbal description for all the pictures taken in the app.

How we built it

We used Node.js to build a web application server that accepts pictures and analyze the objects in them. To achieve this, we used Google cloud platform vision analysis API to analyze the objects inside the photos. Then we did a rhetorical analysis of all the words received and constructed a fluent sentence. We used Google Natural Language processing API to finish the analyzing works. After we constructed a sentence, we used ResponsiveVoice.js to trigger the phone speaker to read it.

Challenges we ran into

  1. compatibility of different browsers and mobile platforms.
  2. Difficulty of finding a suitable model for sentence construction.

Accomplishments that we're proud of

The app is working really well and we deployed it on AWS.

What we learned

Machine learning, product design, fast deployment, multi-platform, AWS

What's next for Eyes

build a native app on IOS and android to have a better user experience. improve the model of sentence construction return the location of objects in the pictures so that the app can report the positions of them.

Share this project: