Beacon

Inspiration

Beacon was inspired by the belief that AI can be a powerful tool to transform lives, particularly for those with visual impairments or physically challenged. Despite some existing tools for assisting, most are either outdated, inefficient, or not user-friendly. We wanted to create a solution that not only works seamlessly but also provides real-time, accurate assistance. Beacon is designed to bridge the gap, offering an advanced AI-powered assistant that empowers users to navigate independently and confidently. This has never been done before at such a high level of integration and accuracy, and the need for such a solution has never been greater.

What it does

Beacon is an intelligent assistant designed to provide real-time navigation support for visually impaired and physically challenged users. Using a smartphone’s camera, it detects objects, recognizes scenes, and performs text recognition while providing voice feedback to the user. With commands like "What's in front of me?" or "Read this sign," Beacon helps users understand their surroundings instantly. Its unique blend of AI models for object detection (YOLOv7), scene recognition (Places365), and Optical Character Recognition (OCR) allows it to process multiple tasks simultaneously and relay accurate, real-time information to the user in a natural, human-like voice. The app also supports intuitive voice commands, making it completely hands-free.

How we built it

We built Beacon by integrating several state-of-the-art technologies:

YOLOv7: This model handles real-time object detection, identifying common objects such as people, vehicles, and obstacles. It’s fast and accurate, perfect for users navigating their surroundings.
Places365: This model was chosen for scene recognition because it’s tailored for indoor environments, providing a perfect solution for users navigating spaces like malls or buildings. For outdoor environments, the system can be easily switched to more specialized models like DeepLab or U-Net, which handle complex, real-world scenes.
OCR (Optical Character Recognition): To assist in reading text from documents, signs, and other sources, we incorporated OCR, allowing users to get real-time verbal feedback on visible text.
Voice Interface: Built with natural language processing (NLP) using AssemblyAI for speech recognition and Eleven Labs for lifelike text-to-speech responses, it enables users to interact with Beacon using voice commands. The project was developed using Python, PyTorch, and OpenCV, and integrates these models to deliver an efficient and seamless experience.

Challenges we ran into

One of the major challenges was integrating multiple AI models to work together in real-time. Synchronizing YOLOv7 for object detection, Places365 for scene detection, and OCR for text recognition was complex. The decision to use Places365 for indoor environments was strategic, as it’s highly specialized for recognizing indoor scenes, but switching to outdoor models like DeepLab or U-Net when needed added a layer of complexity. Additionally, ensuring the voice feedback was both accurate and responsive to the user's queries required careful tuning. Managing model efficiency and balancing the demands of real-time processing with voice accuracy pushed our skills to the limit.

Accomplishments that we're proud of

We're proud of building an eye for the blind — a working prototype of a system that can assist visually impaired users in navigating their environment independently and with confidence. The system's ability to seamlessly integrate multiple AI models for object detection, scene recognition, and text reading in real-time while delivering accurate voice feedback is a huge accomplishment. Moreover, the app's user-friendly voice interaction and hands-free operation elevate it beyond just a functional tool to an indispensable daily aid for visually impaired users.

What we learned

Throughout the process, we learned that time management is key, especially when working on complex, multi-model systems. Things don’t always go according to plan, but our ability to adapt and persevere allowed us to deliver a fully functional system in just 36 hours. We realized the immense potential of our skills and how much of an impact we can make when focused and determined. On the technical side, we honed our skills in PyTorch, OpenCV, and model training and deployment. Most importantly, we learned that it's always about the end user. Understanding their needs drove every decision we made during this project, and it was this mindset that helped us overcome technical challenges and build something truly meaningful.

What's next for Beacon

Beacon has the potential to grow into a full-blown app that goes beyond individual assistance. Future updates could include sending live location updates to loved ones, so family members or caretakers can keep track of the user's whereabouts and ensure their safety. We also plan to enhance the scene recognition model for outdoor environments, refine the accuracy of the voice assistant, and introduce more features for contextual awareness. The long-term goal is to make Beacon a comprehensive navigation assistant that provides emotional support, navigational guidance, and safety alerts, making the world more accessible for visually impaired individuals.