youraibuddy

Features
App main screen
Face attributes recognition
Text recognition
Light detection
How object detection + depth estimation works

Inspiration

It started as an experience. Our friend Manu Prasad is a visually impaired person. We had seen him struggling in day-to-day life. Sometimes he can't navigate well with his stick and he can't use any AI app that helps to read or identify objects due to low light, since he doesn't know whether there is light or not. We just looked every possible android app available on the play store, but none has this feature. He said "If there is no light how can my already installed app can help to identify an object or read text??". During the hackathon, we started working on every possible problem he is facing. We connected with Kerala Blind Association. Mr. Sudheesh, Mr. Umeshan T, Mr. Sathyan, and hackathon mentors who guided us to understand the actual problem the community is facing since three of them are visually impaired and they are members of the blind community and Sathyan sir is a blind school headmaster.

Special Thanks to Mr. Sudheesh for extensive testing of the app

What it does

There is 4 mode in the android app.

Indoor and outdoor navigation with a state-of-the-art hybrid AI depth estimation and object detection model. Help you to hear the object, how far from you. and light detection using the depth estimation model.
Face attribute recognition including age, gender, mustache, beard, sunglasses, eyeglasses, open eyes, and open mouth.
Text recognition (US-English)
AI buddy - chatbot using Facebook AI Research BlenderBot API

How we built it

We build it using python, Kotlin, OpenCV, TensorFlow, etc.

Navigation - We used SSD Mobilenet V2 object detection model for detecting objects, Then Crop the bounding box and apply a Monocular Depth Estimation model on the cropped bitmap for depth estimation and we used a mathematical formula to get the absolute distance.
Light detection- We get the lowest pixel value from the depth inference and applied a threshold to detect light.
Face attributes recognition - Used AWS Rekognition with AWS Lambda and AWS S3, the file gets deleted after processing.
Text recognition - Used AWS Textract with AWS Lambda and AWS S3, the file gets deleted after processing.
Buddy AI chatbot - Used Facebook AI Research team ParlAI language model, we build a custom API for blenderbot 90M model. which is both empathetic and knowledge retrieval model.

Challenges we ran into

Integrating all the solutions as one, was harder than we thought. Introducing the voice response, building and adding the blender bot API to Kotlin, adding the optimized light detection logic, combining the depth estimation model with the object detection model, all pushed our ability to the limit.