We noticed there are no good applications that help blind people learn about their surroundings well. For example, Without the help of their families and friends, they even don't know what is in a photo they just shot.
What it does
This app provides a verbal description for all the pictures taken in the app.
How we built it
We used Node.js to build a web application server that accepts pictures and analyze the objects in them. To achieve this, we used Google cloud platform vision analysis API to analyze the objects inside the photos. Then we did a rhetorical analysis of all the words received and constructed a fluent sentence. We used Google Natural Language processing API to finish the analyzing works. After we constructed a sentence, we used ResponsiveVoice.js to trigger the phone speaker to read it.
Challenges we ran into
- compatibility of different browsers and mobile platforms.
- Difficulty of finding a suitable model for sentence construction.
Accomplishments that we're proud of
The app is working really well and we deployed it on AWS.
What we learned
Machine learning, product design, fast deployment, multi-platform, AWS
What's next for Eyes
build a native app on IOS and android to have a better user experience. improve the model of sentence construction return the location of objects in the pictures so that the app can report the positions of them.