Ocular Sensus

A 3D-printed mockup of the Ocular Sensus device being worn by a person
Ocular Sensus App UI
Architecture diagram showcasing the system's components, including backend, hardware, software, I/O modules, and integrated Google services.
Flowchart for the Ocular Sensus app's emergency framework
Flowcharts detailing the system's logic, environmental analysis, and GPS-guided navigation

Inspiration: Visually impaired individuals face immense challenges navigating daily life, from object identification to safe movement in unfamiliar spaces and lack of proper emergency assistance. Traditional aids often fall short, limiting independence and increasing safety risks. There's a critical need for a solution that enhances mobility, safety, and independence.

What it does: Ocular Sensus addresses core challenges for the visually impaired, it offers environmental understanding, independent navigation, and emergency response. It's an AI-powered wearable with a dedicated app that offers: - Environmental Analysis: Audibly describes surroundings objects, people, and hazards using AI. Navigation (Google Maps Integration): Provides GPS navigation with audio cues using Google Maps Platform, guiding users to their destination. Emergency System: A button sends the user's location to a Firebase database, alerting local responders and, if needed, friends/relatives via SMS. Auto-enables if immobility is detected. Dedicated App: Configures settings and manages emergencies, leveraging Google Maps for real-time location and assistance.

Unlike basic obstacle detectors, Ocular Sensus provides a holistic, AI-powered auditory understanding of the environment and seamless navigation.

How we built it: Ocular Sensus uses a Raspberry Pi as its core, integrating camera, microphone, GPS, compass, speaker, and buttons. Environmental Analysis: Images from the camera are sent to Google Gemini for analysis, and the text output is read aloud via speakers. Navigation: We leverage Google Maps Platform to get directions from the current location to the destination. Audio cues guide the user, including a "compass mode" for initial alignment and continuous updates as they move. Emergency System: The emergency button triggers location sharing to Firebase. The Flutter app constantly checks Firebase for emergencies, displaying helper locations on a map via Google Maps and notifying those within a specified radius.

Key technologies include Google Maps, Google Gemini, Firebase, Google Text-To-Speech, Google Speech-To-Text, and Flutter (Dart).

Challenges we ran into: Integrating diverse functionalities like AI-powered environmental analysis, precise Google Maps navigation, and a robust emergency response into a single wearable presented significant technical challenges. Ensuring seamless real-time data flow between the Raspberry Pi, Google services, and the Flutter app, especially for location-critical features, required intricate design. Developing a crowdsourced assistance system that efficiently connects users in need with nearby helpers via Google Maps and Firebase also posed complexities in backend management and communication protocols.

Accomplishments that we're proud of: We are proud to have been among the GDG Summer Solution Top 14 Finalists. We successfully combined object detection, Google Maps navigation, and emergency alerts into one wearable device. We're particularly proud of the real-time crowdsourced assistance feature, expanding support beyond personal contacts.

What we learned: We learned the importance of a holistic approach for assistive technology, emphasizing rich auditory environmental descriptions over simple obstacle detection for true independence. Integrating Google Maps Platform's comprehensive mapping and navigation capabilities was crucial for precise and intuitive guidance. We also recognized the power of cloud-connected solutions like Firebase for scalability, real-time updates, and multi-language support, essential for global impact.

What's next for Ocular Sensus: Our next steps include further enhancing AI capabilities for even more accurate environmental descriptions and integrating advanced Google Maps Platform features for more dynamic and intuitive navigation. We aim to expand language support and strengthen the emergency response network by onboarding more local responders and potentially integrating with official services. Continued refinement of the modular hardware design and active incorporation of user feedback will also be key to making Ocular Sensus a truly transformative solution globally.