Sarathi

Inspiration

When we were listing out problems that we could potentially solve, our team remembered being lost on our trek back down a mountain, and the feelings of helplessness and fear we had at the time.
We understand that as individuals, we need to make the world inclusive for those with special needs, and we hope this project will scale up to be a dependable app for the visually impaired.

What it does

Sarathi is an easily accessible aid with verbal output for the visually impaired, helping them navigate areas and avoid obstacles in their path according to AR input from the surroundings.

Using AR, Sarathi scans through the camera’s field of vision.

With a single tap, the user is informed verbally about the distance to the nearest obstacle in the path ahead.
With a double tap, the user receives a verbal description of the surroundings in the direction the user is facing.
With a long press for around 5 seconds, a text message is sent containing a rescue message and user’s current location in case of an emergency.

How we built it

Cloning and experimenting with AR related samples was the first step we took towards the development of this product.
To receive the distance to the nearest object in front of the user, we used AR Feature Points to find the nearest feature point and its perpendicular distance to the user.
To receive an audio description of the area in the camera’s field of vision, we used Microsoft's Computer Vision API and Text-to-Speech API to send a snapshot from AR and receive an mp3 with the verbal output.
We used Firebase Storage to receive and store the snapshots from AR.
We also accessed the mp3 files (to play on the iOS device) uploaded from Firebase Functions via the Firebase Storage.
Using the Nexmo SMS Messaging API, we send SMS messages containing a help message and the current known location of the user.

Challenges we ran into

We are a team of backend and Android developers making an iOS app using Apple ARKit for the first time, so there were a lot of places where we met a dead end.
When we tried to implement ARCore on Android, we ran into a lot of unknown errors that we weren’t able to resolve due to documentation issues, and that’s when iOS happened.
None of had worked with iOS before, and it was a challenge as we figured out how to implement the features we had invested our ideas in.
The most time-consuming block we were stuck on was not being able to make specific HTTP requests with files in iOS to Microsoft’s computer vision API.

Accomplishments that we're proud of

Overcame the challenges!
Making the project without prior experience in iOS or AR seemed like a leap that could’ve been too high, but we did end up learning quite a bit of iOS and AR (at least we think so).
Being able to produce and present a working prototype of our idea in two days from scratch is an achievement we do not take lightly.
Our interest in iOS and AR has shot up, and we’re pretty sure there are going to be more projects like this one in our near future.

What we learned

Google and Microsoft seem to have already made products for most of the initial ideas we came up with and that anyone can think of.
Shooting down most of our own ideas because we found they already existed was a damper on our confidence at the beginning of the hackathon.
Ideas that seem cool are the ones that aren’t very implementable or useful and waiting for that one seemingly overused idea with a touch of our individuality makes the perfect idea!
AR opens a world of possibilities to improve how humanity views the world, and we want to learn to be an integral part of it as we move further into our careers.

What's next for Sarathi

The biggest hurdles will be to

Improve accuracy to increase understanding of the surroundings and find relevant elements to the user.
Reduce latency to provide more realtime-like output to the user.

A feature we would love to add to this app is a Find me in a crowd feature.

The user will be able to leave AR Anchors along their path.
Other users can then see the anchors to trace this shared path and find their way.
The user will also be able to send these placed AR anchors in a message to an emergency contact so the user can be easily found.