Scuba AI

set up
Landing page
Live
GCP Proof: Log Explorer
GCP Proof: Dashboard versions
GCP Proof: Builds
GCP Proof: Docker Artifacts

💡 Inspiration

Scuba diving is one of the most thrilling experiences on Earth — and one of the most unforgiving. Divers juggle depth, air supply, ascent rate, navigation, and marine hazards simultaneously, all in an environment where a single mistake can be fatal. We asked: what if your dive computer could see what you see? What if it could read your gauges, identify that fish, warn you about a hazard, and talk you through navigation — all in real time, without you lifting a finger? Scuba.ai was born from the belief that multimodal AI can be the dive buddy that never panics, never gets distracted, and never runs out of air.

🤿 What It Does

Scuba.ai is a multimodal AI dive companion that turns any waterproof smartphone into an intelligent dive computer. It overlays a heads-up display on a live camera feed and uses Google Gemini to analyze frames every two seconds — reading analog gauges, identifying marine species, and detecting hazards like rapid ascent or dangerously low air.

Three specialized AI agents work in parallel:

Safety Agent — reads SPGs, depth gauges, and dive computers via zero-shot OCR; enforces 8 threshold categories with 3-tier severity alerts
Bio Agent — classifies marine life with common/scientific names, safety ratings, and fun facts; auto-flags dangerous species
Nav Agent — analyzes hand-drawn dive maps, extracts landmarks and routes, and provides real-time course corrections using compass data

A Gemini Live voice layer sits on top, triaging agent reports by priority and narrating only what matters — giving the diver a single, calm AI voice. The diver can also speak back: "What's that fish?", "Upload a map", or "Switch to marine biologist mode" — all handled through Gemini's function calling.

The HUD displays real-time dive metrics (depth, air pressure, heading, temperature, bottom time) in a sleek glass-panel interface with priority-coded agent cards — from calm cyan info to flashing red critical alerts.

🛠️ How We Built It

Frontend: React 19 + TypeScript with Vite 6, styled with Tailwind CSS v4 and animated with Motion. The camera feed captures frames via the MediaDevices API, encodes them as base64 JPEG, and routes them through two parallel pipelines — one to the backend agents (every 6s via WebSocket), one directly to Gemini Live (every 2s via the bidirectional streaming API). The HUD was hand-crafted with custom CSS: glass-panel blur, scanline animations, radial vignette, and crosshair reticle — designed to feel like a next-gen dive computer.

Backend: FastAPI + Uvicorn serving a WebSocket endpoint that dispatches frames to the three agents concurrently. Each agent calls Gemini 2.5 Flash with a specialized system prompt and returns structured JSON. Hard-coded threshold checks run after the model responds — the model suggests, the code enforces.

Voice: Gemini Live API with the gemini-2.5-flash-native-audio model provides always-on bidirectional audio. The diver's mic streams 16kHz PCM via an AudioWorklet, and agent reports are injected as text context via sendRealtimeInput — keeping Voice Activity Detection undisrupted. Six function declarations enable voice-controlled settings.

Deployment: Multi-stage Docker build (Node frontend → Python backend) deployed on Google App Engine Flex for WebSocket support.

🧗 Challenges We Ran Into

Structured output consistency — Gemini sometimes narrated instead of returning clean JSON, so we built robust fallback parsing with regex extraction and markdown fence stripping.

Frame capture cadence — Too fast burns API quota and lags the UI; too slow misses critical changes. We landed on a dual-frequency approach: 2s for Gemini Live video, 6s for backend agents.

Voice Activity Detection — Our biggest technical hurdle. Injecting agent reports via sendClientContent() created explicit "user turns" that disrupted VAD, silencing the diver's mic. The fix was switching to sendRealtimeInput({ text }), which streams alongside audio without interrupting detection.

HUD readability — Designing an overlay that's informative but not distracting on unpredictable underwater lighting required extensive iteration on opacity, blur, color, and layout.

Agent chatter — Getting the Live API to stay silent by default (instead of narrating every agent report) required careful system prompt engineering with explicit "SILENCE BY DEFAULT" rules.

🏆 Accomplishments We're Proud Of

Multi-agent priority system — critical hazards get red full-screen overlays with "Immediate Action Required"; routine species IDs get calm cyan styling. The most dangerous condition always surfaces first.
Dual-pipeline architecture — structured backend agents for reliable data extraction + Gemini Live for natural voice interaction, working in harmony without interfering with each other.
Voice-controlled dive computer — six function-calling tools let the diver control the entire app hands-free, which is essential underwater.
The HUD — glass-panel aesthetic with animated depth gauge, compass rose, and scanline effects that genuinely feels like a next-gen dive computer.
Zero-shot gauge reading — the safety agent reads analog SPGs, depth gauges, and dive watches it's never seen before, with threshold enforcement running as a safety net behind the model.

📚 What We Learned

Structured output transforms AI from chatbot to tool — forcing Gemini to respond in typed JSON schemas made the difference between a demo and something usable.
VAD is fragile — any API call that creates an explicit "user turn" will disrupt voice detection. Real-time text injection must use streaming inputs, not content messages.
HUD design is its own discipline — every pixel matters when your UI is overlaid on a live, moving, variably-lit scene. Less is more.
Native audio models have different rules — they only support AUDIO output modality (not TEXT), require outputAudioTranscription for text, and use different model names than the standard Live API docs suggest.
Multi-agent orchestration needs a priority queue — without one, agents talk over each other and overwhelm the diver with information.

🔮 What's Next

Hardware integration — Bluetooth connectivity with dive computers (Shearwater, Garmin) to replace simulated metrics with live sensor data
Dive logging — persistent AI-annotated dive logs with species sightings, route traces, and safety events reviewable after each dive
AR glasses — integration with heads-up display hardware (Meta smart glasses) for a true hands-free experience without needing a phone
Community platform — shared AI-identified species sightings and crowdsourced dive site maps
Offline mode — on-device models for areas without connectivity, with sync when back on the surface