sarathi

Inspiration

We often see blind and visually impaired people struggling while walking on the streets, and there isn't always someone around to help them. That stuck with us. We wanted to build something that could act like a companion for them — something that tells them what's around, so they don't have to depend on someone else being there every time.

How we built it

We used YOLOv8 to detect objects through a camera, then wrote our own logic to figure out the direction (left/right/ahead) and distance (near/far) of each object based on where it appears in the frame. We built a Nepali dictionary ourselves to convert object names into Nepali, and used Google's text-to-speech to speak the final sentence out loud. We wrapped everything in a simple web app so it's easy to run and demo.

Challenges we ran into

We tried integrating an LLM API to make the sentences sound more natural, but we kept running into issues — slow response times, unreliable connections, and inconsistent output during testing. Since this is meant to help someone in real time, we couldn't risk delays or wrong information, so we had to step back and rebuild the sentence logic ourselves instead of depending on an external API. We also lost a lot of time to dependency issues (PyTorch/torchvision mismatches) and discovered that offline text-to-speech doesn't support Nepali well at all, which took a while to figure out.

What's next for Sarathi

We want to properly integrate an LLM this time, done right, so the descriptions feel more natural and detailed instead of template-based. The bigger goal is to move this off a laptop/phone entirely and build it into a wearable — a blind stick or smart glasses with a small camera (like an ESP32-CAM) attached, so a person can use it hands-free while walking, without needing to hold anything at all.

Built With

• computer-vision
• gtts
• natural-language-processing
• opencv
• pygame
• python
• streamlit
• text-to-speech
• ultralytics
• yolov8