1 - RELAY Dashboard
2 - Tech Stack

Relay — Devpost Submission

Category: Live Agents

Inspiration

Learning electronics is intimidating. You're staring at a breadboard, a bag of components you can't identify, and a wiring diagram that makes no sense. YouTube tutorials pause at the wrong moment. Forum answers assume you already know what a voltage divider is. And if you wire something wrong, you don't find out until smoke comes out.

We wanted to build the lab partner we wished we had — someone who watches your bench, talks you through each step, catches mistakes before they happen, and actually gets excited when your LED lights up. Not a chatbot with a text box. A live, voice-driven tutor that sees what you see and responds in real time.

The Gemini Live API made this possible for the first time: bidirectional audio and vision streaming with a model that can genuinely watch and respond to a physical workspace.

What it does

Relay is a real-time voice-and-vision lab assistant for electronics. You point your webcam at your breadboard, and Relay watches, listens, and guides you through building circuits.

The core loop:

Pick a build from the library (e.g. Light-Activated Alarm)
Show your components — Relay verifies inventory via camera
Wire step-by-step — Relay guides placement and catches mistakes in real time
Generate code — Relay writes code sketches and helps debug
Run and validate — Relay watches serial output and confirms the circuit works
Get a 3D digital twin of the completed build in the sandbox

Beyond the live session:

Adaptive quizzes — AI-generated questions that adapt to your skill gaps, with wrong-answer requeue and persistent review history
Drawing exercises — freehand canvas with multi-color pens and eraser where users draw circuit diagrams, submitted to Gemini Vision for AI-powered correctness assessment and targeted feedback
3D Sandbox — interactive breadboard workspace with component placement, wire editing, Monaco code editor, circuit validation, and simulation with real buzzer audio
Community feed — share completed builds, react, and comment (persisted via Firestore)
Progress systems — XP earned from real actions (persisted event log), streaks, leagues, and a draggable knowledge graph
Demo Mode — for judges without hardware: turn on your camera and hold up any object. Relay identifies it in real time, demonstrating the full vision pipeline without needing an electronics kit

Relay has a distinct personality — it's a quirky, enthusiastic engineer who uses electricity puns, celebrates your wins, and has genuine opinions about components. It's not a generic assistant; it feels like a real lab partner.

How we built it

Multi-model architecture: Relay uses a single voice session with gemini-live-2.5-flash-native-audio for real-time audio I/O. Behind the scenes, it dispatches to stronger models via ADK tool functions:

Role	Model
Live tutor (voice + vision)	`gemini-2.5-flash-native-audio`
Quick checks (component ID)	`gemini-2.5-flash`
Code generation	`gemini-2.5-pro`
Deep reasoning (debugging)	`gemini-2.5-pro`
Drawing assessment (vision)	`gemini-2.5-pro`

The user hears one continuous voice. Multiple models work transparently behind the scenes.

Tech stack:

Frontend: React 18 + TypeScript + Vite + Tailwind CSS
3D: Three.js via @react-three/fiber and @react-three/drei (all models procedurally generated, no GLTF imports)
Code editor: Monaco Editor
Live backend: FastAPI + Google ADK with bidirectional WebSocket streaming
Quiz backend: FastAPI + google-genai SDK (adaptive questions + drawing assessment)
Persistence: Firestore REST API
Deployment: Google Cloud Run (two independent microservices)

Google Cloud services:

Cloud Run — live-bridge and quiz-engine as independent microservices
Vertex AI — Gemini model access via service account (no API key needed)
Firestore — user state, XP events, quiz history, community data
Google ADK — agent framework for bidirectional audio/video streaming

Third-party libraries: React, Vite, Tailwind CSS, Three.js, @react-three/fiber, @react-three/drei, Monaco Editor, Lucide React, FastAPI, Pydantic, Uvicorn — all MIT/BSD/ISC licensed.

Challenges we ran into

Gemini Live API session limits are hard constraints. 15 minutes for audio-only, 2 minutes with video active. These are per-session and imposed by the API itself — there's no workaround. We built auto-reconnect with exponential backoff and surfaced the limits in the UI so users understand disconnections aren't bugs.

Native audio models send duplicate messages. The same response arrives as both content.parts text and outputTranscription text — two different wordings of the same thing. Without dedup, every agent response appeared twice. We solved this by only displaying outputTranscription (the actual spoken words) and suppressing content.parts text entirely for audio sessions.

Base64 encoding mismatch broke audio playback. ADK's Pydantic serialization uses URL-safe base64 (- and _ characters), but the browser's atob() only accepts standard base64 (+ and /). Every audio chunk silently failed. We had to normalize the encoding before decoding.

3D in React is heavy. @react-three/fiber + drei add ~1MB to the bundle. We used React.lazy code splitting and extracted shared preset data to separate modules to avoid breaking lazy loading boundaries.

The model hallucinates components. Without explicit guardrails, the vision model would claim to see a breadboard even when the camera was pointed at an empty desk. We added strong anti-hallucination rules to the system instruction requiring the model to only describe what it can actually see.

Adaptive quiz types are unpredictable. The Gemini model sometimes returns question types we didn't build UI for (like spot_error), even when we specify allowed_types. Client-side filtering prevents crashes from unsupported formats.

Accomplishments that we're proud of

Relay actually speaks. This isn't a text chatbot — the Gemini native-audio model produces real voice output, and the frontend plays PCM audio chunks in real time with gapless scheduling. The voice has personality.

Drawing assessment works end-to-end. Users draw circuit diagrams on a freehand canvas with multiple pen colors and an eraser. The drawing is exported as base64, sent to the quiz-engine backend, and assessed by Gemini Vision for correctness — identifying components, checking accuracy, and returning targeted feedback with a confidence score. XP is awarded based on the result.

The 3D sandbox is fully procedural. Every component — Arduino Uno with labeled pins and IC chip detail, breadboard with visible holes, LEDs with glow halos, resistors with color bands, LDR with zigzag pattern, buzzer with polarity marking — is built from Three.js geometry. No imported models. Components are placeable, deletable, and wirable. The simulation validates circuit connections and plays actual buzzer audio through the Web Audio API.

XP is real. Every point of XP is traceable to a specific persisted event: starting a session (+15), answering a quiz correctly (+10), completing a lesson (+25), drawing exercise (+10/+3). The streak auto-updates based on the current date. Nothing resets on page reload — it's all in Firestore.

Multi-model dispatch is invisible. The user talks to one voice, but Flash handles quick lookups, Pro generates code and assesses drawings, and 2.5 Pro does deep debugging. The architecture is clean — each tool function internally calls the appropriate model.

Demo Mode works for judges without hardware. Start a session, skip the electronics, hold up everyday objects to the camera. Relay identifies them in real time and walks through the build conversationally. The full experience is accessible without any physical components.

Architecture

┌──────────────────────────────────────────────────────────────────────────────┐
│                              RELAY FRONTEND                                  │
│                  React 18 + TypeScript + Vite + Tailwind CSS                 │
│                                                                              │
│  ┌────────────┐ ┌────────────┐ ┌─────────────┐ ┌───────────┐ ┌───────────┐ │
│  │   Build    │ │   Learn    │ │  Sandbox    │ │ Community │ │  Library  │ │
│  │ (Live AI)  │ │ (Quizzes + │ │ (3D + Code  │ │  (Feed +  │ │ (Project  │ │
│  │            │ │  Drawing)  │ │  + Simulate)│ │  Social)  │ │ Templates)│ │
│  └─────┬──────┘ └─────┬──────┘ └─────────────┘ └─────┬─────┘ └───────────┘ │
│        │               │                              │                      │
│        │               │         ┌────────────────────┘                      │
│        │               │         │                                           │
│  ┌─────▼──────┐  ┌─────▼──────┐ │  ┌──────────────────────────────────────┐ │
│  │ WebSocket  │  │ HTTP POST  │ │  │     Firestore REST API Client       │ │
│  │ (bidi      │  │ /generate  │ │  │  (user state, XP events, community, │ │
│  │  audio +   │  │ /assess-   │ │  │   quiz history, streaks, leagues)   │ │
│  │  video)    │  │  drawing   │ │  └──────────────┬───────────────────────┘ │
│  └─────┬──────┘  └─────┬──────┘ │                 │                         │
│        │               │        │  ┌──────────────▼──────────┐              │
│        │               │        │  │  Web Audio API (buzzer) │              │
│        │               │        │  │  Three.js (procedural   │              │
│        │               │        │  │   3D components)        │              │
│        │               │        │  │  Monaco Editor (code)   │              │
│        │               │        │  └─────────────────────────┘              │
└────────┼───────────────┼────────┼───────────────────────────────────────────┘
         │               │        │
    ═════╪═══════════════╪════════╪═══════════  NETWORK  ══════════════════════
         │               │        │
    ┌────▼────────┐ ┌────▼────────┐  ┌────────▼──────────┐
    │  LIVE       │ │  QUIZ       │  │   FIRESTORE       │
    │  BRIDGE     │ │  ENGINE     │  │   (Cloud)         │
    │             │ │             │  │                    │
    │  FastAPI +  │ │  FastAPI +  │  │  /users/{id}      │
    │  Google ADK │ │  google-    │  │    ├─ skillProfile │
    │             │ │  genai SDK  │  │    ├─ xpEvents[]  │
    │  Endpoints: │ │             │  │    ├─ streak      │
    │  ws://      │ │  Endpoints: │  │    ├─ quizHistory │
    │  /ws        │ │  /generate  │  │    └─ community   │
    │             │ │  /assess-   │  │                    │
    │  Bidi       │ │   drawing   │  │  Project:          │
    │  streaming: │ │  /health    │  │  relay-gemini      │
    │  PCM audio  │ │             │  │                    │
    │  + video    │ │  Fallback   │  └────────────────────┘
    │  frames     │ │  question   │
    │             │ │  bank when  │
    │  Auto-      │ │  AI unavail │
    │  reconnect  │ │             │
    │  w/ backoff │ │             │
    └──────┬──────┘ └──────┬──────┘
           │               │
      Google Cloud Run  Google Cloud Run
      (independent)     (independent)
           │               │
    ═══════╪═══════════════╪══════════  VERTEX AI  ════════════════════════════
           │               │
    ┌──────▼───────────────▼──────────────────────────────────────────────────┐
    │                        GEMINI MODELS (Vertex AI)                        │
    │                     Project: relay-gemini                               │
    │                     Auth: Service account (no API key)                  │
    │                                                                         │
    │  ┌─────────────────────────┐  ┌─────────────────────────────────────┐  │
    │  │  LIVE BRIDGE MODELS     │  │  QUIZ ENGINE MODELS                 │  │
    │  │                         │  │                                     │  │
    │  │  gemini-2.5-flash       │  │  gemini-2.5-pro                     │  │
    │  │   -native-audio         │  │   → Question generation             │  │
    │  │   → Always-on voice +   │  │   → Drawing assessment              │  │
    │  │     vision streaming    │  │     (Gemini Vision)                 │  │
    │  │                         │  │                                     │  │
    │  │  gemini-2.5-flash       │  │  Fallback: pre-built question bank  │  │
    │  │   → Quick component ID  │  │   (no model needed)                 │  │
    │  │   → Fast Q&A            │  │                                     │  │
    │  │                         │  └─────────────────────────────────────┘  │
    │  │  gemini-2.5-pro         │                                           │
    │  │   → code gen    │                                           │
    │  │   → Code debugging      │                                           │
    │  │   → Deep reasoning      │                                           │
    │  └─────────────────────────┘                                           │
    └─────────────────────────────────────────────────────────────────────────┘

Data flow — live session:

User speaks → Mic → PCM audio → WebSocket → Live Bridge → ADK → Gemini Flash Native Audio
                                                                         │
Gemini responds ← ADK ← Live Bridge ← WebSocket ← PCM audio chunks ←───┘
         │
         ├── Voice output → Web Audio API (24kHz PCM playback, gapless scheduling)
         ├── Transcription → Chat transcript (outputTranscription only, deduped)
         └── Tool calls → Dispatched to 2.5-Flash/2.5-Pro as needed

Data flow — drawing assessment:

User draws on canvas → base64 PNG → POST /assess-drawing → Quiz Engine
    → Gemini 2.5 Pro Vision → { correct, feedback, components, confidence }
    → Frontend displays result → XP event fired → Firestore persistence

What we learned

Multimodal dedup is a real engineering problem. When a model produces audio, text, and transcription simultaneously, you need explicit rules about what to display. "Show everything" creates a confusing duplicate experience. "Show only transcription" is clean but requires detecting the response modality.

Procedural 3D is underrated. GLTF models are great for static scenes, but when users need to place, move, and wire components dynamically, procedural geometry gives you full control. The tradeoff is more code, but the result is a lighter bundle and complete flexibility.

Anti-hallucination in vision requires explicit rules. Vision models will confidently describe things that aren't there. "Only describe what you actually see" isn't enough — you need "If you haven't received camera frames, say you can't see anything." Specificity beats general instructions.

Drawing assessment adds a powerful learning modality. Asking users to draw circuits — not just answer multiple choice — forces deeper understanding. Gemini Vision's ability to identify hand-drawn components and evaluate correctness makes this viable without any custom ML model.

What's next for Relay — Real-Time Voice & Vision Lab Tutor for Electronics

Full circuit solver. The sandbox validates component presence, wiring, and connection structure — but a proper SPICE-style solver that computes actual voltages and currents across nodes would enable real-time electrical feedback as users build.

More builds. The Light-Activated Alarm is the flagship, but the architecture supports any project. Temperature monitors, motor controllers, IR remotes — each just needs a preset definition and build instructions.

Longer sessions. As Gemini Live API limits expand, Relay's value increases proportionally. A 15-minute guided build session is genuinely useful. A 60-minute session could cover an entire project from concept to completion.

Bonus Points

1. Published Content

Blog post covering how Relay was built with Gemini and Google Cloud, created for the purposes of entering this hackathon:

Medium: I Built a Real-Time AI Lab Partner for Electronics Learning with Gemini Live

2. Automated Cloud Deployment

Deployment is fully automated via deploy.sh in the public repository.

./deploy.sh              # Deploy all services to Cloud Run
./deploy.sh live-bridge  # Deploy live-bridge only
./deploy.sh quiz-engine  # Deploy quiz-engine only
./deploy.sh verify       # Health-check deployed services

The script handles GCP project configuration, deploys both backend microservices (live-bridge and quiz-engine) to Cloud Run with all required environment variables, builds the frontend, and verifies service health — all in a single command.