We often don’t realize how different the world is for disabled people, such as the blind and the deaf. To raise awareness, we wanted to create a technology that can both assist and empower them for a very long time. Since some of us have relatives who are blind, we decided to create an app that can take pictures or images, analyze what is in that photo, and read that response out loud. Vision-impaired people can thus use this app to scan their surroundings, read signs and papers without Braille, and freely explore their surroundings.

What it does

VisualEyes is a hybrid app that works on both web and mobile. A user can either take pictures themselves or upload photos on their camera gallery, and the app will immediately say out loud what is in that picture. It is able to detect people, objects, and scenery, making it robust and description. The user can also use VisualEyes to read street signs, store names, and handwriting, so they are able to read books and papers without relying on Braille. In addition, the website has a location heatmap feature which graphically shows where the user takes the most pictures. A history feature displays the most recent images.

How we built it

We used Javascript, Jquery, and a Bootstrap template for the front end, and used Hasura for the backend services such as nginx and Docker. For the analysis part, we used the Microsoft Computer Vision API to take in images and output both a description of what is in the image, and also if the image contains any words. The app then uses Microsoft’s Cognitive Services API and text-to-speech function to read the descriptions out loud. The images are uploaded to Hasura with its File Storage API, and we also used Hasura's database (which is built on Postgres) to hold strings for our image urls and descriptions. We use AJAX requests to get the image URLs and descriptions for populating the "Picture History" section and also used the Google Maps API to track a user's geolocation of where they take the most pictures.

Challenges we ran into

Implementing the APIs was a challenge for us, especially when we were working with Hasura, because it was everyone’s first time using the API. It’s a relatively new technology and so we often had to ask mentors for help (shoutout to all the helpful mentors!). We also spent a lot of time trying to decide on what to do in the beginning of the hackathon deciding how we would turn our idea into a working prototype. So along with programming, we spent quite a bit of time doing independent research and utilizing the mentors to help us answer our questions.

Accomplishments that we’re proud of

We’re very proud of seeing how VisualEyes grew from a small idea to a full-fledged project. We’re also very proud of the fact that we learned how to use the Hasura API and Microsoft Cognitive Services API with the help of some amazing mentors, in such a short amount of time.

What we learned

We learned that while it is quite simple to come up with a list of ideas, it’s difficult to pick just one to commit to, especially if there was more than one idea that we really liked. We also learned some amazing new skills including how to use new technologies such as the Hasura API and the Google Maps API.

What’s next for VisualEyes:

We hope to keep refining the mobile side and making it more responsive. We would also want to incorporate camera functionality to be able to get descriptions of an environment in real time.

Built With

Share this project: