Gemi-Eye – Proactive Guardian for the Visually Impaired

Eyes that never blink. Memory that never fades. A heart that understands how you feel.

Comment

Inspiration

Gemi-Eye is a proactive multimodal AI guardian for visually impaired users, built with Gemini Live and deployed on Google Cloud, capable of continuously observing the environment, reasoning in real time, and speaking only when safety or assistance truly requires it. More than 43 million people worldwide live with blindness. Every day they navigate a sighted world that was never designed for them — stepping into kitchens with forgotten knives, crossing busy streets filled with moving vehicles, and searching rooms where important objects vanish into clutter.

Most assistive technologies today are reactive. They wait for a question:

“What is in front of me?” “Where is my phone?” “Is something dangerous nearby?”

But danger rarely waits for a question.

I wanted to build something fundamentally different — an AI that acts before being asked. An AI that continuously observes the environment, remembers important details, warns of danger instantly, and comforts users during moments of distress — like a trusted companion who never sleeps.

That idea became Gemi-Eye — a proactive AI guardian powered by Gemini Live’s real-time multimodal intelligence. Unlike traditional assistive tools that wait for commands,Gemi-Eye represents a new class of AI systems: proactive multimodal agents that continuously perceive the world and autonomously decide when to act.

What it does

Gemi-Eye is a real-time multimodal AI companion that continuously observes the user’s environment, reasons about what it sees, and decides when it should speak, warn, remember, or stay silent.

The entire system is voice-first, meaning visually impaired users never need to interact with screens or text.

Proactive spatial awareness

Using real-time camera streaming, Gemi-Eye detects objects and describes them using an intuitive clock-face spatial model.

Example:

“Water bottle at your 9 o’clock, about 1.5 meters away.”

Unlike traditional assistants, Gemi-Eye can speak without being asked when something important appears.

Instant safety guardian

Dangerous objects such as knives, scissors, or open flames trigger immediate spoken warnings.

Example:

“Warning. Knife at your 2 o’clock, approximately 0.3 meters away. Please be careful.”

These alerts occur within seconds, even if the user has not issued a command.

Persistent spatial memory

Gemi-Eye remembers objects and their last known positions using a persistent spatial memory system.

This allows the system to answer questions such as:

“Where did I leave my phone?”

—even hours or days later.

Emotional intelligence

The system analyzes voice tone and language patterns to detect distress.

If a user says:

“I’m scared.”

Gemi-Eye shifts its tone to provide calm and supportive reassurance.

Caregiver network

A companion dashboard allows caregivers or family members to receive:

-proactive safety alerts

-environmental observations

-live reasoning traces

This provides peace of mind for loved ones who want to ensure the user’s safety.

How we built it

Gemi-Eye is built as a real-time multimodal AI agent powered by Gemini Live, enabling native streaming of voice and vision simultaneously.

Gemini Live enables true bidirectional multimodal interaction, allowing the system to process camera frames and voice input simultaneously while generating natural spoken responses.

Core architecture

Perception Layer

Camera frames stream continuously at 1280×720 resolution.

Gemini Vision analyzes each frame to detect objects and estimate spatial relationships. These detections are converted into clock-face positions and distance estimates, which are easier for blind users to interpret.

The system processes environmental updates multiple times per second, allowing safety alerts to trigger in under three seconds.

Context Engine

A continuously updated context model combines:

-visual observations

-voice input

-spatial memory

-conversation history

-silence duration

This provides a dynamic understanding of the environment over time.

Reflection Oracle v7 (Decision Engine)

Instead of hardcoded rules, the system sends the full context to Gemini, which returns a structured JSON decision:

-speak

-interrupt

-notify caregiver

-remain silent

This enables true autonomous agent behavior.

Persistent Memory System

Detected objects are stored in a SQLite spatial database containing:

-object type

-position

-confidence score

-timestamp

This memory persists across restarts.

Safety & Interrupt Engine

Dangerous objects trigger priority voice interruptions, ensuring warnings are delivered immediately.

Emotional Intelligence Engine

Voice tone and contextual keywords detect distress and trigger supportive responses.

Caregiver Dashboard

A monitoring dashboard streams alerts and reasoning through WebSockets, allowing caregivers to monitor important events.

Cloud Infrastructure

The system is deployed on Google Cloud to ensure reliable real-time processing and scalability.

Core services include:

-Vertex AI + Gemini Live API for multimodal reasoning and streaming interaction

-Cloud Run for hosting backend agent services

-WebSocket streaming layer for low-latency communication

-Cloud Storage for logs and buffered media

-Secure API gateway powering the caregiver dashboard

This architecture allows Gemi-Eye to process multimodal streams in real time while maintaining persistent context across sessions.

Challenges we ran into

Building a real-time multimodal AI system introduced several engineering challenges.

Real-time streaming stability

Gemini Live connections occasionally triggered keepalive timeouts (1011 errors) during long sessions.

We solved this by implementing proactive reconnection cycles and heartbeat signals every 55 seconds.

Memory reliability

Early versions accidentally deleted useful memories during initialization.

We redesigned the filtering logic so over 100 environmental memories now load reliably at startup.

Interrupt reliability

A bug involving an undefined epoch variable caused occasional interrupt failures.

Fallback logic was added to ensure warnings always trigger correctly.

Silence detection overflow

Long sessions caused silence timers to overflow.

Resetting timestamps during initialization resolved the issue.

Emotion detection accuracy

Early sentiment detection was inconsistent.

We improved accuracy using confidence thresholds and contextual keyword detection.

Accomplishments that we're proud of

-Built a proactive AI guardian prototype in under two weeks

-Achieved real-time safety alerts in under three seconds

-Implemented persistent spatial memory across restarts

-Designed a voice-first system tailored for visually impaired users

-Built a caregiver dashboard for remote monitoring

-Demonstrated a true multimodal autonomous agent architecture

What we learned

Building Gemi-Eye revealed that autonomous agents are fundamentally different from chatbots.

A chatbot simply answers questions.

An autonomous agent must constantly decide:

-Should I speak now?

-Is this important enough to interrupt?

-Would silence be better?

Balancing helpfulness and intrusion was one of the most difficult design challenges.

We also learned that real-time multimodal AI systems require careful engineering to remain stable while processing streaming voice, vision, and reasoning simultaneously.

Most importantly, we learned that accessibility is not just a feature — it is a design philosophy.

What's next for Gemi-Eye – Proactive Guardian for the Visually Impaired

Next steps include:

-Mobile deployment on Android and iOS

-Integration with smart glasses and wearable cameras

-Offline fallback mode for unreliable internet environments

-Deeper long-term emotional memory

-Community-driven improvements to object recognition accuracy

Gemi-Eye demonstrates the potential of AI systems that watch over users rather than waiting to be asked.

For someone who cannot see the world around them, even a small warning can make a life-changing difference.

Technology should not merely answer questions.

Sometimes, it should simply be there — watching, remembering, and speaking when it matters most.

For someone who cannot see the world, that moment of awareness can mean safety, independence, and confidence.

Built With

Share this project:

Updates

Submission history