Developing an application for social good definitely inspires us, especially when we are building an application that allows us to support users who are visually impaired. Based on the facts from World Health Organization (WHO), an estimated population of 253 million people live with vision impairment, 36 million are blind and 217 million have moderate to severe vision impairment. Majority of this population are from a developed country where technology is easily assessable. With a boon in the domain of computer vision, we decided to make our mobile device’s camera as the eyes for these users to recognize objects in their surrounding. The app can also be used by the kids as a learning tool during early childhood.

What it does

clickAndKnow is an Android application that utilizes a mobile device’s camera to record a video. The user can tap on the screen when they want to recognize an object.The app selects the frame at that instance and speaks out the object identified in that frame.

How we built it

We have made use of the Camera2 API that starts the camera in the device and enables users to start recording the surrounding objects. When the user taps on the screen, it selects the frame at that instance and speaks out the object identified in that frame. The object recognition is done using Microsoft Cognitive API. The application reads the the identified object’s name displayed using TextToSpeech android library.

Challenges we ran into

Understanding the user needs to develop a user-friendly as well as assessable application

Accomplishments that we're proud of

Serving the committee with the medium of technology.

What we learned

Integration of the Microsoft Cognitive API in our code and using Android Studio.

What's next for clickAndKnow

Utilizing wearable device’s camera in order to get a hands-free experience

Built With

Share this project: