IRIS - INTEGRATED REAL-TIME INTERPRETIVE SYSTEM

Inspiration

Millions of visually impaired individuals struggle to navigate everyday environments independently. We wanted to build something that acts as a third eye — an always-on companion that can see and describe the world in real-time. The Gemini Live API's ability to process video and respond with natural audio made this vision possible.

What we learned

The Gemini Live API has two distinct input channels — sendRealtimeInput() for continuous video/audio streaming and sendClientContent() for text messages. Understanding this architectural difference was critical to reducing hallucination.
Real-time vision-language models still struggle with grounding — describing only what is actually visible without fabricating details.
Client-side scene detection using pixel-diff comparison with a mean absolute difference formula:

$$MAD = \frac{1}{N} \sum_{i=1}^{N} |P_i^{current} - P_i^{previous}|$$

is a lightweight way to trigger model responses only when the environment actually changes, reducing unnecessary output.

How we built it

Iris is built on React with TypeScript, using the @google/genai SDK to connect to the Gemini 2.5 Flash Native Audio Preview model via WebSocket. The webcam feed is captured at 50% resolution, 3fps as JPEG frames and streamed through sendRealtimeInput(). Audio is recorded at 16kHz PCM via an AudioWorklet.

For scene change detection, we implemented a pixel-diff algorithm on an $80 \times 45$ downscaled canvas — when the grayscale mean absolute difference exceeds a threshold $\tau = 20$ with cooldown $\Delta t = 8000\text{ms}$, the model is prompted to describe the new scene.

Text inputs are paired with the current video frame as inline data to maintain visual context across both API channels. The frontend is deployed on Google Cloud Run using a Docker multi-stage build with Nginx.

Challenges we faced

Hallucination — The preview model frequently fabricated objects not present in the scene. We iterated through $6+$ system prompt versions, tuned frame resolution from $0.25$ to $0.5$ scale, and added grounding constraints to minimize this.
Auto-polling failure — Our initial approach of periodically prompting the model caused constant hallucination because the audio output modality forces a response for every sendClientContent() call. We replaced this with pixel-diff detection where a scene change triggers only when $MAD > \tau$.
Text vs voice input disparity — Text messages had no visual context because they use a different API channel than the video stream. We fixed this by capturing and attaching the current video frame as inlineData to every text message.