Personal AI Operator: Gemini 2.0 Live Voice Agent

Real-time voice AI using Gemini 2.0 Live API with intelligent turn-taking. Features Voice Activity Detection, native audio streaming Pure voice conversation without text dependency

Comment

Inspiration

The challenge was to move beyond text-based interactions with AI. I wanted to create a system where users could have natural, real-time conversations without typing or reading text. Most voice AI systems suffer from overlapping speech - both user and AI talk simultaneously. I decided to build something that truly understands conversation flow.

What I Learned

  1. Voice Activity Detection (VAD) is critical for natural conversations. A 2-second silence detection threshold proved optimal for Urdu/English mixed speech.

  2. WebSocket architecture with proper handshaking prevents audio pipeline race conditions. The VOICE_READY signal must complete before sending audio chunks.

  3. Gemini 2.0 Live API requires strict PCM 16-bit, 16000Hz audio format. Even 24kHz causes silent failures - no errors, just data drops.

  4. AudioWorklet processing is essential for real-time streaming without blocking the main thread.

  5. Turn-taking complexity: The system must handle silence detection, buffer management, and proper turnComplete signaling simultaneously.

How I Built It

  • Frontend: React + Vite with AudioWorklet for real-time audio capture
  • Backend: Node.js/Express with WebSocket relay to Gemini 2.0 Live API
  • Infrastructure: Google Cloud Run for scalable deployment
  • Audio Pipeline: PCM 16-bit encoding, RMS-based silence detection, turnComplete events
  • Language Support: Native Urdu/English support via Gemini's multilingual capabilities

Challenges Faced

  1. Handshake Race Condition: Frontend was sending audio before backend confirmed session readiness. Fixed with explicit voiceReady state tracking.

  2. Silent Disconnects: Model name prefix mismatch (models/ was missing) caused Gemini to silently crash. Added timeout protection and proper error logging.

  3. Audio Format Mismatch: Browser's native sample rate was 24kHz but Gemini only accepts 16kHz. Required explicit AudioContext initialization at 16000Hz.

  4. RMS Threshold Tuning: Initial threshold of 0.015 was too high, dropping legitimate speech. Lowered to 0.005 for better sensitivity.

  5. Tools Deadlock: Using tools and responseModalities together in native-audio mode caused setupComplete to never trigger. Removed tools temporarily for stability.

Outcome

A production-ready voice AI that demonstrates proper multimodal interaction without text dependency. The system maintains natural conversation flow with intelligent turn-taking and handles edge cases like overlapping speech, network delays, and audio quality variations.

Built With

Share this project:

Updates

Submission history