Hank Home Inspection AI

App analyze water heater model

## Inspiration

Home inspectors examine hundreds of components per inspection — water heaters, electrical panels, HVAC units — and they're expected to catch every defect, read every label, and know every recall. It's a high-stakes, hands-full job where missing one corroded fitting or one recalled appliance model can mean real danger for a homeowner.

We asked: what if an inspector could just point their phone camera at a piece of equipment and have an AI copilot that sees what they see, reads labels they'd otherwise squint at, and knows whether that model has been recalled — all in real time, hands-free, by voice?

That's HiHank's Live Agent: a Gemini Live-powered copilot built into a mobile inspection app that turns a phone camera into an AI-augmented pair of eyes.

## What We Built

HiHank's Live Agent streams the inspector's camera feed and microphone audio to Google's Gemini Live API in real time. The inspector talks naturally — "What brand is this water heater?" or "Check if this model has any recalls" — and Gemini responds by voice while analyzing the live camera frames.

But it goes beyond Q&A. Gemini can directly manipulate the inspection checklist through a tool-calling system: creating checklist items, logging observations, recording defects — all by voice command with a confirmation step so the inspector stays in control.

The architecture has three layers:

React Native mobile app (Expo) — captures 16kHz PCM audio and JPEG camera frames, streams them over WebSocket
WebSocket relay on Cloud Run — bridges the mobile client and Gemini Live API, transforming message formats bidirectionally and routing tool calls back to the backend for execution
Next.js backend — manages sessions, validates permissions, executes tool calls against the Supabase database, and enforces inspector/organization data boundaries

## How We Built It

The core challenge was real-time bidirectional streaming across three systems (phone → relay → Gemini) with tool execution in the loop.

On the mobile side, we built audio capture with ~128ms batched chunks to balance latency against network overhead, and configured iOS audio sessions for simultaneous microphone input and speaker output — so the inspector hears Gemini's response while the mic stays ready.

The WebSocket relay on Cloud Run acts as a protocol bridge: it translates our client message format into Gemini's realtimeInput format for audio/images and clientContent for text, then parses Gemini's streaming responses back into typed messages (audio, text, tool calls, turn signals). When Gemini invokes a tool — like create_observation to log a defect — the relay forwards it to our backend API, waits for the result, and feeds it back to Gemini so the conversation continues seamlessly.

We used Supabase with Row Level Security for all inspection data, ensuring that tool calls from the AI respect the same organization-level access controls as the rest of the app.

## Challenges

Audio echo and interruption handling. When Gemini speaks, its own audio was being picked up by the phone mic and fed back, causing loops. We solved this by pausing audio capture while Gemini is in the speaking state — a simple state machine gate that eliminated the echo.

WebSocket lifecycle management. Bridging two independent WebSocket connections (client and Gemini) means either side can drop at any time. We implemented heartbeat pings every 15 seconds, idle timeouts at 120 seconds, and graceful session-end hooks so inspection data is never left in a dirty state.

Tool call confirmation UX. Letting an AI modify real inspection data is powerful but risky. We added a confirmation flow: when Gemini wants to create or modify a checklist item, the mobile app shows the proposed change and waits for the inspector to approve before executing. This kept the AI useful without making it dangerous.

Latency vs. reliability tradeoff. Direct WebSocket streaming is fast but fragile on mobile networks. We tuned audio buffer thresholds (~150ms before flushing to the speaker) and message batching to keep the conversation feeling natural even on spotty cellular connections.

## What We Learned

Building a real-time multimodal AI agent is less about the AI and more about the plumbing — audio encoding, WebSocket state machines, protocol translation, and graceful degradation. The Gemini Live API is remarkably capable once you get clean audio and images to it; the hard part is everything between the phone microphone and the model.

We also learned that tool-calling with confirmation is the right pattern for AI agents that touch real data. Fully autonomous agents are scary in professional contexts; copilots that propose and confirm are trusted.