How I Built ItThe project is built on a "Multi-Agent Coordinator" architecture using the Google Agent Development Kit (ADK).The Voice Core: I utilized the Gemini 2.0 Flash Live model to handle bidirectional audio streaming, allowing for natural "barge-in" where a doctor can interrupt the AI during a case review.The Vision Engine: Since clinical software often lacks DOM access, I built a Voice UI Navigator tool. It captures screenshots of the imaging software and uses Gemini 2.0 Flash to identify UI elements and pathological markers directly from pixels.Real-time Grounding: To ensure diagnostic accuracy, the agent is grounded via the Google Search tool, allowing it to cross-reference findings with the latest clinical guidelines.Infrastructure: The backend is containerized and deployed on Google Cloud Run, utilizing Secret Manager for secure API key handling and Artifact Registry for CI/CD.Challenges FacedThe primary challenge was managing token accumulation during long diagnostic sessions, as native audio tokens accumulate at approximately $25$ per second. I implemented ContextWindowCompressionConfig to maintain performance. Another hurdle was separating concerns: I learned that for stability, the live audio session should not handle vision analysis simultaneously; instead, it triggers a separate synchronous tool call to analyze medical artifacts.What I LearnedI discovered that Gemini's multimodal reasoning can improve diagnostic accuracy by up to $18.51\%$ when used as an assistant. Most importantly, I realized that AI doesn't need deep software integration to be useful—it just needs to "see" what the doctor sees.
Built With
- artifact-registry
- cloud-logging.-apis/tools:-gemini-live-api-(websockets)
- gemini-2.0-flash.-frameworks:-google-agent-development-kit-(adk).-languages:-python-3.11.-cloud:-google-cloud-run
- models:-gemini-2.0-flash-live-(gemini-2.0-flash-live-001)
- search
- secret-manager
Log in or sign up for Devpost to join the conversation.