• 

  EyeFriend

• 

  Real-Time Multimodal Interaction

• 

  Architecture

EyeFriend

Features and Functionality

EyeFriend is a real-time voice-driven accessibility assistant built for visually impaired and blind users. Using the device camera and microphone, it continuously perceives the user's environment and narrates it aloud, so the user never needs to look at or touch the screen. The assistant delivers four capabilities in parallel: it proactively detects hazards and obstacles and speaks warnings without waiting for a prompt; it reads visible text such as signs, labels, and menus aloud; it generates periodic scene summaries when the user enters an unfamiliar space; and it produces tactile-style overhead maps of the environment on voice command, described verbally as a spatial layout the user can mentally navigate. Alert delivery is tuned to avoid overwhelming the user, a built-in 8-second cooldown deduplicates repeated warnings, while urgent hazards such as moving vehicles or open stairs bypass the cooldown and interrupt immediately. Every session is logged to Cloud Firestore, tracking alert and text-read counts, and an analytics endpoint aggregates statistics across sessions so usage patterns can be studied over time.

Technologies Used

The entire intelligence layer is powered by Google Gemini. The primary model, gemini-2.5-flash-native-audio-preview-12-2025, processes interleaved PCM audio and JPEG video natively in a single bidirectional stream via the Gemini Live API, accessed through the official @google/genai JavaScript SDK. This native multimodal capability eliminates a separate transcription or text-to-speech pipeline and is what makes sub-second response latency possible. Camera frames are sent at 4 FPS alongside 16 kHz PCM microphone audio; the model returns 24 kHz PCM audio responses played back via the Web Audio API. Beyond speech, the model issues structured function calls that drive UI state independently of the audio stream:

// Three tool declarations sent to Gemini at session start
tools: [{ functionDeclarations: [
  proactiveAlertDecl,    // real-time hazard warnings
  sceneUpdateDecl,       // environment descriptions
  tactileMapDecl         // overhead spatial maps
]}]

Tactile maps are generated on demand by a second model, gemini-2.0-flash-preview-image-generation. An optional Python-based server-side agent uses Google Agent Development Kit (ADK) with gemini-2.0-flash-live-001, exposing a WebSocket endpoint built with FastAPI and Uvicorn as an alternative streaming path:

agent = LlmAgent(
    model="gemini-2.0-flash-live-001",
    name="eyefriend_agent",
    tools=[proactive_alert, update_scene_summary, generate_tactile_map],
)

Google Cloud Infrastructure

All backend services run on Google Cloud. The Node.js REST API and the React frontend are deployed as separate services on Cloud Run, configured to scale to zero when idle. Docker images are stored in Artifact Registry, and session data is persisted in Cloud Firestore in native mode, accessed via Application Default Credentials without explicit key management. The backend runs under a dedicated IAM service account scoped to roles/datastore.user only, following the principle of least privilege. The entire infrastructure is provisioned declaratively with Terraform using the Google Cloud provider and deployed end-to-end by a single shell script with no manual console interaction:

export GOOGLE_CLOUD_PROJECT=your-project-id
export GEMINI_API_KEY=your-key
bash deploy.sh
# Builds images, pushes to Artifact Registry, runs terraform apply

The frontend is a React 19 and TypeScript 5 single-page application bundled with Vite 6, designed as a Progressive Web App so it can be embedded in any platform without native packaging. The high-contrast black and yellow color scheme follows RNIB accessibility guidelines, and typography uses Helvetica Neue based on low-vision legibility research.

Data Sources

EyeFriend does not rely on any pre-collected dataset, labeled corpus, or third-party data feed. All inference runs in real time against live camera and microphone input from the user's device, and no images or audio are stored beyond the active Gemini session. The system prompt governing model behavior was written by hand and iteratively refined, drawing on WHO guidelines on visual impairment, WCAG 2.2 accessibility standards, and Google's responsible AI practices.

Findings and Learnings

The most significant finding during development was that native audio output transforms the user experience. An early prototype piped a text-output model into browser TTS; switching to the native audio model with voice "Charon" produced responses that were 300–400 ms faster and noticeably more natural, a difference that matters when the user is navigating in motion. Google AI Studio was essential for rapidly prototyping the streaming configuration before writing production code, and the Gemini Live API reference provided the detail needed to handle bidi-stream lifecycle correctly.

The hardest engineering problem was managing backpressure when sending video frames and audio simultaneously over a single WebSocket: incoming frames must queue without blocking audio playback, and profiling with Chrome DevTools was necessary to eliminate audio dropout under load. The inbound stream bandwidth per second is approximately \( B = F \cdot S_f + R_a \cdot D_a \), where \( F \) is frame rate, \( S_f \) is frame size in bytes, \( R_a \) is audio sample rate, and \( D_a \) is bit depth in bytes. For EyeFriend's configuration this works out to:

$$ B = 4 \times S_f + 16000 \times 2 $$

Under mobile CPU constraints this requires deliberate throttling to avoid audio dropout.

Proactive tool calling, where the model autonomously decides when to issue an alert rather than waiting for a user prompt, proved more reliable than expected once the system prompt was tuned using Google's prompt engineering guide and system instruction best practices. On the infrastructure side, wiring the backend Cloud Run URL into the frontend Docker image at build time via a Terraform output reference was a subtle but important detail, documented in the Cloud Run Terraform resource reference. Finally, Google ADK reduced the server-side agent to roughly 150 lines of application logic by abstracting session management, tool dispatch, and streaming lifecycle, a significant simplification compared to managing raw WebSocket frames manually.

Built With

• gcp
• gemini