Gemini Live Agent Challenge

live Agent Seamless Mic, Camera , Text input and Ouput Voice , Text and Video Generate Features Agent Core live Video Any Thing Ask Response LLM backend Gemini and Any other Models Optional

Comment

Inspiration

285 million people worldwide are blind or low-vision. They navigate a world built for sighted people — reading labels, avoiding obstacles, understanding surroundings — all without seeing. We asked: what if AI could be their eyes, in real-time, with just a phone camera and voice? That question became NurAI (نور — Light in Urdu) — an AI companion that gives light to those in darkness.

What it does

NurAI is a real-time multimodal live agent powered by Google Gemini. It takes any input and responds in any form — seamlessly:

Core Inputs:

  • Live Microphone — speak naturally, ask anything, interrupt anytime (barge-in supported)
  • Live Camera/Video — continuous video stream analyzed in real-time (scene understanding, hazard detection, text reading)
  • Text Input — type questions when voice isn't an option

Core Outputs:

  • Live Voice Response — instant, natural spoken answers with low latency
  • Text Transcript — real-time conversation transcript on screen
  • Proactive Alerts — warns about hazards (stairs, obstacles) without being asked

Live Agent Features:

  • Seamless Live Video Q&A — point camera at anything, ask any question, get instant spoken answer
  • Barge-in/Interruption — speak mid-response and the agent stops, listens, responds to your new question
  • Multilingual — English and Urdu voice support
  • Conversation Memory — remembers context across the session
  • Natural Personality — calm, compassionate, concise responses

All powered by Gemini 2.0 Flash Live model with native audio + vision streaming through Google ADK.

How we built it

  • LLM Brain: Google Gemini 2.0 Flash Live (gemini-2.0-flash-live-001) via Google ADK — the only model with native real-time audio + vision streaming
  • Agent Framework: Google Agent Development Kit (ADK) with LiveRequestQueue for bidirectional streaming
  • Camera: OpenCV captures frames at 1fps → JPEG bytes sent to Gemini in real-time
  • Audio: sounddevice for mic input (16kHz PCM) and speaker output (24kHz PCM from Gemini)
  • UI: Gradio web interface — start/stop button, live transcript, language toggle
  • Backend: Python async pipeline — three concurrent coroutines (camera, mic, response playback)
  • Deployment: Docker container on Google Cloud Run with public HTTPS URL

Challenges we ran into

  • Handling real-time audio + video streaming simultaneously required careful async coordination with asyncio.gather
  • Implementing barge-in/interruption — detecting when the user speaks during playback and immediately stopping TTS output
  • Keeping latency low while sending camera frames — balanced at 1fps to avoid Gemini rate limits while staying responsive
  • Cross-platform audio setup — switched from PyAudio to sounddevice to avoid C compiler dependency issues
  • Managing Cloud Run cold starts for a real-time demo — used --min-instances=1 to keep it warm

Accomplishments that we're proud of

  • Built a fully working real-time voice + vision agent in 2 days
  • True barge-in interruption — feels like talking to a real person
  • Proactive hazard detection without being asked — the agent watches and warns
  • Reads text in Urdu and English from camera — practical daily use for blind users
  • Deployed to a public Cloud Run URL anyone can try
  • Named نور (Light) — gives light to those who live without sight

What we learned

  • Gemini 2.0 Flash Live API is uniquely powerful for real-time multimodal agents — no other model streams audio + vision natively
  • Google ADK simplifies live agent development significantly with LiveRequestQueue and RunConfig
  • Real-time AI requires thinking in streams, not request-response — every component must be async and non-blocking
  • Accessibility-focused AI has massive untapped potential — 285M+ people need this today

What's next for NurAI

  • Mobile app — native Android/iOS app with always-on camera and voice
  • Navigation mode — GPS + camera fusion for outdoor walking directions
  • Object memory — "remember where I put my keys" with spatial tracking
  • More languages — Arabic, Hindi, Spanish, French
  • Offline mode — on-device Gemini Nano for basic scene description without internet
  • Wearable integration — smart glasses with bone conduction audio for hands-free use

Built With

Share this project:

Updates

Submission history