GozAI

• 

  architecture_diagram

• 

  logo

Inspiration

For the 2.2 billion people experiencing vision impairment globally, standard assistive technology often feels like a burden rather than a bridge.

The seed for GozAI was planted in 2022 during my 500-level Optometry externship at the University of Ilorin Teaching Hospital (UITH) in Nigeria. Working under a low-vision specialist, I witnessed a frustrating reality: traditional aids are fragmented, expensive, and stigmatizing. Patients were paying a premium for clunky, single-purpose magnifiers. It raised a glaring question: We are in the digital age. Why hasn't this been solved using the devices already in our pockets?

Recent 2026 clinical research completely validates this, showing a staggering abandonment rate of up to 50% for specialized assistive gadgets due to cost, battery drain, and social stigma. People just want their smartphones to act as their eyes.

With the era of agentic coding and the Gemini Live Agent Challenge, the technical barriers vanished. I set out to build a multimodal accessibility copilot that is battery-optimized, emotionally reassuring, and rooted entirely in clinical reality.

What it does

GozAI is an empathetic, voice-first companion acting as the eyes and hands for people with low vision. It tackles the Activities of Daily Living (ADLs) that dictate a patient's independence.

We engineered four powerful modes:

Scene (Live Agents Track): Runs continuously via camera streaming. It detects hazards, gives spatial audio descriptions ("Stairs ahead"), and uses synchronous haptic feedback. Crucially, it supports genuine barge-in interruptions—a safety necessity, not just a feature.

Screen (UI Navigator Track): The ultimate Digital Accessibility Bridge. Where standard screen readers fail on unlabelled buttons or OS pop-ups, GozAI calculates coordinates and injects synthetic taps for the user (e.g., "Goz, click 'Allow'").

Read: Conversational document reading powered by Gemini vision and offline ML Kit OCR. Perfect for sensitive medical labels or restaurant menus.

Companion: Launched via zero-UI voice commands ("Hey Siri, open Goz"). Goz is clinically tuned to reduce anxiety and act as a supportive talk-partner.

How we built it

We engineered GozAI using a Google-native stack:

Cognitive Engine: Gemini 2.0 Flash Multimodal Live API via bidirectional WebSockets. This streams audio and camera frames concurrently with zero transcription delay.

Backend Agent (Cloud Run): Google ADK Agent deployed to Cloud Run with 4 registered tools: Optometry Guidelines, Medication Lookup, SOS, and Clinical Stats.

Data Layer (Firestore): Firebase isn't just a database; the Firebase MCP toolset runs inside the ADK Agent, acting as an agent-native interface to power Companion Memory and trigger caregiver SOS alerts in under one second.

Frontend (Flutter): We built a dual-aesthetic UI. The Patient Experience uses Brutalist-accessibility (matte black, high-visibility yellow), while the Pro Dashboard acts as a data-dense interface for caregivers.

Challenges we ran into

The biggest challenge was combatting Cognitive Fatigue. A 2025 study in Translational Vision Science & Technology proved that continuous, uncurated audio feedback creates massive sensory overload.

Our Solution: The Semantic Vibro-Acoustic Strategy. Instead of babbling constantly, GozAI acts as an intelligent filter. It processes frames at a battery-saving 1 FPS (dropping to 0.2 FPS when resting via accelerometer data). Gemini uses multimodal reasoning to trigger synchronized haptic pulses (an aggressive pulse for hazards, or a warm haptic pulse every 15 seconds for a clear path).

Accomplishments that we're proud of

We are incredibly proud of grounding our AI strictly in Clinical Reality. We didn't just cite research—we physically wired it into Gemini’s system prompt:

Active Contrast Enhancement (AMD Protocol): We instructed Gemini to actively detect dark scenes and aggressively describe drop-offs that blend into the background.

Clinical Empathy (Glaucoma Protocol): Vision loss carries a 2.48x higher suicide risk. Gemini’s persona is strictly constrained to be calm, steady, and reassuring, acting as an anchor of psychological safety.

Zero-Latency Clinical Data: We hardcoded optometry guidelines into our backend RAG, ensuring Gemini has 0ms-latency access to deterministic, peer-reviewed safety constraints even during network throttling.

What we learned

We learned that the true power of the Gemini Multimodal API isn't just in describing things—it’s in reasoning about when NOT to speak. Building GozAI reinforced the belief that the best accessibility tech reduces friction rather than just adding data. We also experienced the immense power of the Google ADK in transforming complex architectural loops into clean, cloud-native deployments effortlessly.

What's next for GozAI

We plan to deepen GozAI's offline capabilities and scale to Android native AccessibilityService for OS-wide "Ghost Touch" control. We will also release caregiver SDKs so optometry clinics can tap into GozAI's encrypted telemetry, making remote patient monitoring a reality.

Built With

• dart
• firebase
• flutter
• gemini-api
• google-adk
• google-cloud-run
• python
• websockets