Saathi

Inspiration

India has around 5 million blind individuals and 70 million visually impaired individuals, and yet there is no suitable app to assist them. That's when we came up with Saathi. 

Saathi is an AI powered, fully voice activated android application designed to assist the visually impaired community in their daily activities. Our vision is to combine all the features required for daily and repetitive tasks of a blind individual in a single application.

What it does

Saathi leverages the power of cutting-edge technologies like Mulit - Modal Large Language Models (LLMs), Natural Language Processing, Text-to-speech and Text-to-speech models , Computer Vision and Image Processing models and much more. Saathi aligns with the united nations sustainable development goal of reducing inequalities in the society.

Supports Fully voice-activated as well as regular button based app navigation. The UI is minimalistic and easy to use for the visually impaired individuals.

Saathi has 6 different modules : Authentication Module : Provides a fully-voice activated login and signup experience for the blind individual, which has been developed using firebase number based authentication.

LOOK Module : Utilizes the Gemini Pro Vision API which describes the image that is clicked by the user. This module can be used for Outdoor navigation, signboard description, informational or direction board description, learning and much more.

READ Module : Can read any page from a newspaper, book, document, etc. It utilizes Text-Recognition V2 API from the ML-Kit Library.

LEARN Module : Here we utilize the Gemini Pro API to help the blind individuals learn and enhance their knowledge by Voice-based Question Answering. This module can be effiently utilized to enhance knowledge in academic subjects, general knowledge and genral curiosity.

FIND Module : Unlike the look and read module, the FIND module continously scans for common keywords in the surrounding as text format eg : School Bus, Toilet, etc. Thus helping the blind individual to navigate effectively.

Obstacle Detection Module : We wish to build a custom obstacle detection and alerting module which can detect potholes, cars, etc to help the blind inviduals in outdoor navigation. We are planning to combine this with Google Maps for effective travelling.

How we built it

Saathi uses the following technologies:

Kotlin - Programming Language for Android Android Studio - Official Integrated Development Environment (IDE) for Android app development Jetpack Compose - Android UI developent Toolkit ML-Kit -Machine Learning Package for android development Gemini API - Cloud APIs for LLM and Vision Firebase - Back-end and Database Figma - Ui Design and prototyping

Challenges we ran into

We had started this project before the launch of Gemini and had no idea how to implement the Look feature. That's when Gemini was launched at that time, and we could use it in our project. The next challenge was testing our app. We successfully tested it with the help of a local blind school, and they were impressed by it.  We got feedback to make it talkback friendly that was the one challenge we faced but took the challenge and the app works smoothly with talkback now.

Accomplishments that we're proud of

When we tested the look feature, it gave us an accurate description. When we tested our app in schools for special children, they were really fascinated by it, and their smiling faces filled us with immense happiness and joy.

What we learned

While developing the app, we came across a lot of new things that we thought of implementing in the app. We were able to learn the CameraX Library, and then we understood how to implement different versions of Gemini in Android, like Gemini-Pro in the learn module and Gemini-Pro-vision in the look module. We also learn how to integrate AI modules into Android apps. The most important thing to learn was to develop the app with accessibility in mind as people with impaired vision, color blindness, use Android devices. When you develop apps with accessibility in mind, you make the user experience better for people with accessibility needs. So learning the principles to make app more accessible was a something new for and to keep in mind for future projects.

We saw how blind individuals interact with mobile phones and how we can improve our AI so they can easily use our app.

What's next for Saathi

We are continuously testing the app and getting feedback from the visually impaired individuals and making the changes accordingly.

Built With

• android-studio
• firebase
• gogole-cloud
• google-gemini
• jetpackcompose
• kotlin
• ml-kit