BlindSpot

BlindSpot Logo
Start Screen
Main Page

Inspiration

BlindSpot was inspired by a personal experience that highlighted the daily challenges faced by visually impaired individuals. One of our team members witnessed a visually impaired person with complete blindness who was stuck in a corner on the McMaster campus. After offering assistance, he guided them to their residence and learned along the way about their day-to-day activities, how they get to classes and go out to meet with friends with no assistance. Being struck by this realization of how visually impaired people navigate their world, we recognized that modern, urban environments are often not designed with accessibility in mind, especially for the visually impaired.

With crowded streets, construction zones, and busy intersections, cities can present significant challenges for individuals who rely on walking sticks and other mobility aids to navigate. As part of the Best Smart Cities Hack, we wanted to address a real-world problem related to Sustainable Cities and Communities (SDG 11), specifically tackling accessibility issues for the visually impaired.

What it does

BlindSpot is an accessibility-first navigation app designed to help visually impaired users move through urban environments safely and confidently. Using real time GPS data, BlindSpot transforms regular turn-by-turn directions into an AI voice agent that converses and delivers instructions when they are most useful in over 32 languages! The AI voice agent announces upcoming turns early to give users time to prepare and then provides clear, immediate instructions at the exact moment an action is required. This timing based approach reduces confusion, lowers stress, and allows users to stay oriented without needing to rely on visual cues. Moreover, BlindSpot also begins each route with a spoken summary and destination confirmation so users can form a mental map of their journey before navigation starts. Users can also speak to the AI voice agent during any part of their journey for additional information or clearer instructions. In addition, when the camera detects the visually impaired person will run into an object/obstacle, the AI voice agent will alert them verbally and also send haptic feedback through the phone. Thus, by removing the need to interact with a screen and prioritizing non-visual feedback, BlindSpot supports independent mobility and empowers users to navigate the city with confidence.

How we built it

BlindSpot was built as a modular, real-time assistive navigation system centered around a mobile application developed in Flutter. The current implementation has been developed and validated on Android, with the app accessing native device capabilities such as the camera, microphone, location services, and haptic feedback.

Real-time communication between the mobile app and AI services is handled using LiveKit, which serves as a low-latency backbone for streaming audio and visual data. The live camera feed is processed using the Gemini Live API, enabling real-time interpretation of visual input to understand the user’s immediate surroundings. This vision pipeline operates independently from location-based logic to maintain clarity and reliability.

Voice interaction is supported through the device microphone, with spoken input routed through LiveKit to the Gemini API. Gemini acts as a reasoning layer to interpret user commands and generate structured interactions with the Google Maps API, supporting dynamic, voice-driven navigation.

To deliver clear and accessible audio feedback, responses are converted to speech using ElevenLabs, providing natural-sounding voice guidance and supports 32 different spoken languages.

Challenges we ran into

Object Detection: Implementing reliable object detection proved difficult, as it required precise calibration for recognizing obstacles and determining their proximity in real-time. We had to carefully balance responsiveness and accuracy to make the system intuitive for users.

Haptic Feedback: Implementing haptic feedback proved challenging on iOS due to strict system limitations and reliance on predefined vibration patterns. Android offered greater flexibility and direct access to vibration hardware, allowing us to build and test the haptic cues more reliably. As a result, we focused on Android for the initial implementation and demo of BlindSpot’s haptic navigation features.

UI/Flutter: Integrating text-to-speech functionality within Flutter posed minor challenges as we had to learn the necessary syntax and libraries to get it working smoothly. We also faced difficulty with merging the UI development branch into the master branch, but it was sorted out.

Accomplishments that we're proud of

We are extremely proud of the accomplishments we achieved throughout this hackathon, especially the impact our solution has on a vulnerable demographic in our communities. It is both easy to use and practical for everyday activities. Not only did the project work, but we are also proud to have successfully implemented a wide range of technologies we were unfamiliar with, such as an AI voice agent, Flutter, and LiveKit, which allowed us to bridge all APIs and UI together into one cohesive application. Going into the hackathon, all team members were strangers to one another, but we quickly aligned as a team, delegated roles, and made significant progress. We are truly happy with what we were able to build.

What we learned

Through a workshop with GitCloud, we learned how to build and integrate an AI voice agent and connect it to external services. We also gained hands-on experience with Flutter, developing a functional mobile application and learning how to work with device features such as the camera, microphone, and haptics. Additionally, we learned how to use LiveKit as a real-time communication layer to interconnect multiple APIs and manage low-latency audio and data flow across the system.

What's next for BlindSpot

Our next steps include adding a push-to-talk command system so users can issue clear voice instructions on demand without continuous listening. We also aim to improve camera detection speed for more responsive real-time feedback, and introduce a richer set of haptic patterns to convey different types of signals and urgency levels more intuitively.