clickAndKnow

Initial screen
Capturing a video frame
Object Recognition

Inspiration

Developing an application for social good definitely inspires us, especially when we are building an application that allows us to support users who are visually impaired. Based on the facts from World Health Organization (WHO), an estimated population of 253 million people live with vision impairment, 36 million are blind and 217 million have moderate to severe vision impairment. Majority of this population are from a developed country where technology is easily assessable. With a boon in the domain of computer vision, we decided to make our mobile device’s camera as the eyes for these users to recognize objects in their surrounding. The app can also be used by the kids as a learning tool during early childhood.

What it does

clickAndKnow is an Android application that utilizes a mobile device’s camera to record a video. The user can tap on the screen when they want to recognize an object.The app selects the frame at that instance and speaks out the object identified in that frame.

How we built it

We have made use of the Camera2 API that starts the camera in the device and enables users to start recording the surrounding objects. When the user taps on the screen, it selects the frame at that instance and speaks out the object identified in that frame. The object recognition is done using Microsoft Cognitive API. The application reads the the identified object’s name displayed using TextToSpeech android library.