The 21st Century gave us full self-driving cars and yet there’s nothing substantial that can help a blind person navigate their environment independently. Worldwide, 285 million visually impaired and 39 million blind people are affected by this lack of attention to their basic needs. The U.S. alone has 13 million visually impaired and 1.3 million blind Americans. By 2050, global blindness is estimated to reach 115 million and over 550 million cases with visual impairment. Despite the existing range of assistive technology for the visually impaired on the market today, there is no solution that combines functionality and affordability. At EyeLead, we want to give millions of people the chance to lead their lives independently. Using computer technologies we decided to improve the lives of the visually impaired and deliver the most futuristic & interactive visual aid assistant yet.

What it does

We want to create a smart hierarchical approach that will not only classify a scene to the user, but provide enough information to make a decision or be able to make an additional detailed request. Our primary USP is the user’s ability to interact with EyeLead and receive information just like they would if they had help from a sighted human assistant. EyeLead works by scanning the user’s surroundings in real-time and describing the objects in the environment based on the user’s distance from the objects or user’s request. Once EyeLead is initiated it asks the user out loud: "Are you looking for something specific?" You see, EyeLead works by scanning the user's environment and describing their surroundings based on the user's request or distance. Let me explain. If the user answers, Yes, it then asks the user what they are looking for, and based on the object the user names, EyeLead will group the surrounding objects. After the user vocalizes what they are looking for, in this case, "a bed", EyeLead localizes the objects in the same category and then tell the user where the bed is, how far it is from the user, and whether there are any obstacles in the way to the bed. However, if the user said "No," meaning they are not looking for anything specific, EyeLead will scan their environment and describe it in general based on the user's proximity.

How we built it

Our application differs from other navigational applications that assist blind and visually impaired people in that our application does not require users people to memorize finger tapping patterns to use the app. Instead, our app utilizes text-to-speech and speech-to-text APIs to make communication with the app completely auditory. The layout of our app has very few buttons and is relatively easy to use compared to other apps. Other apps require time and memorization for visually impaired people to learn how to use them and we wanted to simplify these apps.

Challenges we ran into

A challenge we ran into was getting the separate object detection application to be properly connected to our main application. Crashes and non-responsiveness were some issues related to this problem and we were able to solve this. Another challenge we ran into was using version control initially.

Accomplishments that we're proud of

Two members of the development team were fairly new to programming coming into this hackathon, so we have grasped many things in different areas that we are proud of. We learned more about the languages XML, Kotlin, and Java which we never quite used before. We were also were very new to app development in general and utilizing APIs, especially with Microsoft azure. We were able to integrate text-to-speech and speech-to-text with azure services and APIs as functional pieces within our app. Additionally, we were able to generate object recognition in our app as well. Throughout the many hours of coding and researching, our biggest accomplishment was learning about azure services and making them work within our app given such a small period to do so.

What we learned

We learned many things. We learn how to implement cloud services into an application. We learned how to build an Android application using XML, Kotlin, and Android Studio. We learned how neural networks in the form of TFLite can be used for easy, mobile AI solutions.

What's next for EyeLead

We plan on further improving our prototype. Our object detection app does not currently respond to specific items requested by the user. Instead, it requests all relevant objects in its view. Additionally, we would like to implement text-to-speech to read out these objects for the user. Furthermore, we would like to learn how to implement depth perception into our project and expand our development and design team to create a working prototype that we can show to our potential customers and receive their feedback for improvement. Our ultimate goal is to make the EyeLead app a reality and to do that we would like to grow our team and potentially partner with Microsft to help with our development.

Built With

Share this project: