• 

  Command Center

Gemini Live Agent Challenge

Submission Focus

I built Aerivon as a real-time multimodal voice agent that users can talk to naturally, interrupt instantly, and steer mid-conversation without breaking flow.

What Inspired Me

I was inspired by a practical frustration: most AI assistants still force users into typed, turn-by-turn chat. In real life, people speak, pause, interrupt, change their mind, and expect the system to adapt.

What specifically inspired each part:

• Live voice interaction: Inspired by seeing non-technical users struggle with typing-heavy interfaces when speech would be faster.
• Interruption handling: Inspired by real conversations where waiting for a bot to finish is unacceptable.
• Visual context support: Inspired by the need for agents that can understand what users are looking at, not just what they type.
• Creative output support: Inspired by product demo workflows where teams need text, visuals, and video quickly from one request.
• Reliability engineering: Inspired by prior brittle systems that failed under noisy audio and unpredictable model outputs.

How I Built The Project

I built a cloud-hosted, realtime agent stack designed for low-latency interaction and interruption safety.

Architecture

• Backend: FastAPI with WebSocket endpoints for live bi-directional streaming.
• Frontend: Svelte UI for mic input, transcript feedback, and streamed voice playback.
• Models: Gemini Live API for real-time multimodal dialogue, plus Veo for media generation workflows.
• Cloud: Google Cloud Run for serving and Cloud Build for automated deployment.

Build Process

1. Built realtime WS channels for speech/text input and streaming output.
2. Implemented hands-free turn-taking and explicit interrupt signaling.
3. Added automatic barge-in logic so speaking can stop assistant playback without manual interrupt clicks.
4. Improved audio playback pipeline with chunk queueing and sample-rate consistency.
5. Added pause-noise suppression and chunk-boundary smoothing to reduce hiss/pop artifacts.
6. Added production validation scripts for voice, roundtrip audio, and websocket health checks.
7. Deployed and verified repeatedly on Google Cloud endpoints.

Challenge Requirements Mapping

Live Agents Requirement

• Natural voice conversation: Implemented.
• Graceful interruptions: Implemented via manual and automatic interrupt flow.
• Real-time response: Implemented with streaming WS pipeline.

Mandatory Tech Requirement

• Uses a Gemini model: Yes (Gemini Live API).
• Uses Google GenAI SDK or ADK: Yes (Google GenAI SDK).
• Uses Google Cloud service(s): Yes (Cloud Run, Cloud Build, Cloud Storage-related workflows).

Key Challenges I Faced

• Barge-in reliability: Ensuring user speech could consistently preempt assistant audio in real time.
• Audio pacing quality: Fixing slow/unnatural playback by improving chunk handling and rate propagation.
• Pause artifacts: Suppressing loud background sound during silence windows.
• WebSocket robustness: Preventing turn-state drift across interrupts and reconnect boundaries.
• Model response variability: Hardening parsing and fallback logic for diverse response shapes.

What I Learned

• A great Live Agent is mostly orchestration quality, not just model quality.
• Latency and conversational control are equally important UX metrics.
• Audio UX quality depends on details: buffering, chunk order, gating, and sample-rate consistency.
• Production testing reveals issues that local testing does not.
• Continuous deploy-test-fix loops are the fastest path to a stable realtime agent.

Technical Lens (LaTeX)

To reason about responsiveness, I modeled end-to-end latency as:

$$ T_{\text{experience}} = T_{\text{input}} + T_{\text{inference}} + T_{\text{stream}} + T_{\text{playback}} $$

And optimized for:

$$ \min\; T_{\text{experience}} \quad \text{subject to} \quad Q_{\text{audio}} \ge Q_{\text{target}} $$

This helped me avoid regressions where speed improved but conversational audio quality degraded.

Final Reflection

This challenge pushed me to design for real human behavior, not scripted demos. Aerivon now behaves like a true live assistant: it listens, responds quickly, handles interruptions, and remains usable in messy real-world interactions.

Built With

• ai
• cloud
• docker
• fastapi
• gemini
• live
• multimodal
• playwright
• python
• run
• vertex
• websockets