Oncydra: Voice-First AI Cancer Care Companion

Voice-first AI that helps cancer patients track symptoms, understand their journey and advocate for themselves, while giving clinicians real-time insights for proactive care.

Comment

🎗️ Oncydra Hackathon Journey

Inspiration

Cancer patients consistently tell us the same things:

"I left the appointment more confused than when I arrived." "I don't know if this symptom is normal or dangerous." "I forgot to ask the questions that mattered most."

The reality is stark: oncology consultations average just 15-20 minutes, yet patients leave with life-altering information they're expected to remember, process, and act on, often while dealing with treatment side effects, anxiety, and fear.

As a PhD student in cancer genomics, I spend my days immersed in algorithms, computational pipelines, and bioinformatics packages. It's easy to get lost in the mathematics of mutation signatures and pathway analysis and forget that behind every dataset is a person navigating the hardest chapter of their life. We optimize models, but seldom do we build tools that directly touch patient care.

I wanted to step out of my comfort zone. Move beyond the computational and into the clinical. Being in a health-focused institution surrounded by clinicians and medical researchers gave me the perfect opportunity to bridge that gap, to take everything I know about AI and apply it to help patients feel heard, informed, and empowered.

Oncydra is voice-first by design, because when you're exhausted from treatment, the last thing you want to do is type. It's not here to replace the care team; it's here to help patients show up to appointments more prepared, track how they're feeling between visits, and ensure nothing falls through the cracks. A bridge between clinic days, not a replacement for them.

What it does

Oncydra is a comprehensive voice-first oncology care platform that supports patients and clinicians throughout the cancer treatment journey.

Patient Features:

  • 🎙️ Daily Voice Check-ins: Patients speak naturally about how they're feeling. The ElevenLabs-powered voice agent asks follow-up questions, extracts symptoms, tracks severity over time, and identifies concerning patterns, all through conversation.

  • 🩺 "Is This Normal?" Triage: When patients worry about a symptom, they can ask Oncydra for context. The AI explains whether the symptom is commonly expected for their diagnosis and treatment phase, but always encourages patients to contact their care team if they remain concerned using ElevenLabs prompt guardrails. Oncydra provides information, not medical advice.

  • 📅 Voice-Powered Timeline Management: Patients add appointments, scans, and treatment dates simply by speaking. "I have a CT scan next Tuesday at St. Vincent's" becomes a structured calendar event, synced to Google Calendar.

  • 🎓 Personalized Learning Hub: AI-generated education content tailored to each patient's diagnosis, treatment phase, and recent symptoms, powered by Google Search grounding for medical accuracy.

  • 💪 Advocacy & Rehearsal Hub: Before appointments, Oncydra suggests questions patients might want to ask based on their recent symptoms and concerns. Patients can practice voicing these questions through conversation with an AI, not to replace the real appointment, but to help them walk in feeling more prepared and confident.

  • 📞 AI-Powered Care Team Calls: When patients need to reach their care team, Oncydra initiates outbound AI voice calls via ElevenLabs' native Twilio integration. The AI knows the patient's profile, diagnosis, treatment phase, recent symptoms, so it can advocate on their behalf. When an appointment is agreed, it's automatically added to their timeline and synced to Google Calendar.

Clinician Features:

  • 👨‍⚕️ Patient Overview Dashboard: Healthcare providers see real-time symptom trends, risk alerts, treatment timelines, and check-in transcripts, enabling proactive care rather than reactive crisis management.

  • 🏥 Consultation Intelligence: Clinicians record or upload consultation audio. AI extracts clinical entities (diagnoses, medications, follow-ups, red flags), generates SOAP notes, and creates patient-friendly summaries, all as drafts requiring, clinician approval before any action is taken.

  • ✅ Human-in-the-Loop Approval: Every AI-generated clinical output (SOAP notes, patient summaries, follow-up tasks) goes through a gatekeeper. Clinicians review, edit, approve, or reject before anything reaches the patient or medical record. Safety flags highlight content needing urgent review.

  • 📊 Population Analytics: BigQuery-powered dashboards show symptom trends across patient cohorts, helping clinicians identify patterns and allocate resources proactively.

How I built it

🏗️ Architecture

Backend Services:

  • 7 Cloud Run microservices + 1 Cloud Function handling distinct domains (symptoms, timeline, education, advocacy, communication, reminders, consultation, analytics)
  • 26+ specialized AI agents orchestrated using Google Agent Development Kit (ADK)
  • Real-time data flow via Firestore listeners and PubSub events

ElevenLabs Integration:

  • 5 Conversational Agents for distinct voice experiences (check-in, triage, timeline, summary, mock rehearsal)
  • 4 Custom webhook tools connecting ElevenLabs agents to Cloud Run backends
  • ElevenLabs Speech Synthesis: Generates realistic consultation demo audio (doctor-patient dialogue) for the clinician transcription feature
  • Native Twilio Integration: Outbound AI phone calls to care teams

ADK Tools:

  • 10 Custom FunctionTools: ADK FunctionTools wrapping Firestore, BigQuery, Gmail, Calendar, and Pub/Sub operations:
  • Google Search grounding: Education agents use ADK's built-in Google Search tool to retrieve accurate, up-to-date medical information from sources like American Cancer Society, NCI, and Mayo Clinic

🤖 Agent Architecture

Our multi-agent system implements advanced agentic patterns; Sequential, Parallel Fan-Out/Gather, Review/Critique, and Human-in-the-Loop:

Symptom Intelligence Service (7 agents) — Parallel + Sequential + Review Patterns

  • SymptomRootAgent (SequentialAgent): Orchestrates the pipeline
  • SymptomExtractionAgent (LlmAgent): Extracts symptoms from natural conversation (parallel)
  • TrendAnalyzerAgent (LlmAgent): Compares current symptoms to historical patterns (parallel)
  • RiskScorerAgent (LlmAgent): Flags clinical risk signals (parallel)
  • SynthesisAgent (LlmAgent): Aggregates parallel results into unified analysis
  • QualityReviewerAgent (LlmAgent): Critique/refine pattern for accuracy validation
  • TriageAgent (LlmAgent): Standalone agent for "Is This Normal?" queries

Timeline Service (1 agent)

  • TimelineExtractor: Extracts appointments, scans, and treatment dates from voice transcripts

Education Service (5 agents) — Sequential Pattern + Google Search Grounding

  • EducationOrchestrator (SequentialAgent): Coordinates the pipeline
  • TopicGenerator: Identifies relevant education topics from patient context
  • ContentResearcher: Uses Google Search tool to retrieve accurate medical information
  • ContentSynthesizer: Assembles personalized education snippets
  • QuickEducator: Standalone single-agent fallback with Google Search

Advocacy Service (4 agents) — Sequential Pattern

  • AdvocacyOrchestrator: Coordinates preparation workflow
  • QuestionCoach: Generates personalized questions for appointments
  • RehearsalDialogue: Creates practice conversation scripts
  • AppointmentPrep: Provides practical preparation information

Reminder Service (4 agents) — Loop Pattern

  • FollowUpGuardian: Orchestrates iterative reminder checks
  • CheckInMonitor: Detects missed patient check-ins
  • AppointmentMonitor: Identifies upcoming and recent appointments
  • ReminderComposer: Generates personalized reminder emails

Consultation Service (5 agents) — HITL Gating Pattern

  • ConsultationOrchestrator: Coordinates clinical analysis pipeline
  • ClinicalEntityExtractor: Extracts diagnoses, medications, treatments, red flags
  • SummaryGenerator: Creates SOAP notes and patient-friendly summaries
  • ConcernDetector: Identifies patient confusion, anxiety, unmet needs
  • HITLGatekeeper: Safety layer blocking actions until clinician approval

Agent models are selected based on task complexity and cost efficiency, Gemini 3 Pro for services requiring advanced clinical reasoning, with Gemini 3 Flash, Gemini 2.5 Flash and Gemini 2.5 Pro available for high-volume, lower-complexity tasks.

Each ElevenLabs agent connects to our Cloud Run backends via custom webhook tools, enabling real-time patient data retrieval and appointment booking.

Google Product Integration:

☁️ Google Cloud Stack:

  • Vertex AI: Powers ADK agents with Gemini models (gemini-2.5-flash, gemini-3-pro-preview & gemini-3-flash-preview)
  • Cloud Run: Hosts 7 microservices (symptom, timeline, education, advocacy, communication, reminder, consultation)
  • Cloud Functions: Analytics service for population-level insights
  • BigQuery: Symptom trend analytics and cohort-level data aggregation
  • Firestore: Real-time patient data, check-ins, timeline events, and consultation records
  • Cloud Pub/Sub: Event-driven architecture for checkin-completed and timeline-updated events
  • Cloud Scheduler: Automated reminder triggers for patient follow-ups
  • Secret Manager: Secure storage for API keys and credentials
  • Google Agent Development Kit (ADK): Multi-agent orchestration with Sequential, Parallel, and custom patterns

📧 Google Workspace / APIs:

  • Gmail API: Email notifications to care teams with branded HTML templates
  • Google Calendar API: Appointment sync and timeline management

🛠️ Google Developer Tools:

  • Google AI Studio: Initial app prototyping and wireframe generation
  • Google Antigravity: Agentic IDE used for development
  • Google Search (via ADK): Grounded medical information retrieval for education content

Challenges I ran into

🎯 Voice UX for Vulnerable Users: Designing voice interactions for cancer patients required exceptional care. I iterated extensively on agent personas, pacing, and responses to ensure conversations felt supportive rather than clinical or robotic.

⚡ Multi-Agent Coordination: Orchestrating 26+ agents across 7 services while maintaining data consistency was complex. I leveraged ADK patterns (Sequential, Parallel, Review/Critique, Fan-Out/Gather), Firestore listeners, and Pub/Sub events to keep agents synchronized across the platform.

🔒 Healthcare Sensitivity: Balancing helpful AI assistance with appropriate medical disclaimers required careful prompt engineering. Oncydra supports and informs, it never diagnoses or replaces professional medical advice. Every clinical output goes through HITL gating before reaching patients.

🔗 ElevenLabs Tool Integration: Connecting ElevenLabs agents to Cloud Run backends required building custom webhook tools that fetch patient context from Firestore, book appointments to timelines, and handle edge cases, all while maintaining natural conversation flow.

🎭 Dual-Interface Design: Building for two very different users, patients who need warmth and simplicity, and clinicians who need density and efficiency meant designing parallel experiences that share the same data layer but feel completely different.

Accomplishments I'm proud of

🏆 Technical Achievements:

  • Built a production-ready multi-agent system with 26+ specialized agents powered by 10 custom FunctionTools across 7 services
  • Implemented advanced ADK patterns (Sequential, Parallel Fan-Out/Gather, Review/Critique, Human-in-the-Loop)
  • Created 5 unique ElevenLabs voice agents with custom webhook tools connecting to Cloud Run backends
  • Achieved real-time symptom tracking with Firestore listeners, Pub/Sub events, and BigQuery analytics

💜 Human Impact:

  • Designed a system that genuinely helps cancer patients feel heard and supported
  • Built advocacy tools that empower patients to speak up in appointments
  • Created clinician tools that enable proactive rather than reactive care
  • Bridged my computational background with real patient-facing care

🎨 User Experience:

  • Voice-first design that's accessible when patients are too exhausted to type
  • Beautiful orbital timeline visualization for treatment journey
  • Seamless Google Calendar integration for appointment management
  • Dual dashboards tailored to patients and clinicians with shared real-time data

🎓 Personal Growth:

  • Shipped a full-stack healthcare platform as a solo developer while balancing PhD research
  • Stepped outside my comfort zone, from bioinformatics pipelines to patient-facing product design
  • Learned the full Google Cloud + ElevenLabs ecosystem from prototyping to deployment

What I learned

🤖 Google Agent Development Kit: ADK gave me a structured way to orchestrate complex multi-agent workflows. Chaining agents, running parallel fan-out/gather patterns, and using lifecycle hooks unlocked a level of control I didn't know was possible.

🎤 ElevenLabs Conversational AI: Building 5 distinct voice personas taught me how much tone and pacing matter when users are vulnerable. Custom webhook tools turned simple voice agents into context-aware assistants that actually know who they're talking to.

☁️ Google Cloud Ecosystem: Firestore listeners, Pub/Sub events, and Cloud Run created a reactive architecture where changes propagate instantly. It shifted my thinking from request-response to event-driven design.

💜 Healthcare AI Design: The biggest lesson wasn't technical, it was learning when AI should step back as ethical AI is very important. Oncydra informs and supports; the care team leads.

What's next for Oncydra

🔮 Immediate Enhancements:

  • Multi-cancer expansion: Extend beyond breast cancer to lung, colorectal, and hematologic malignancies
  • Clinician voice features: Voice dictation for notes, voice-driven patient summaries
  • Predictive risk modeling: Machine learning models for toxicity onset and hospitalization risk using longitudinal symptom data

🚀 Long-term Vision:

  • EHR Integration: Deep sync with hospital systems for seamless clinical data flow
  • Caregiver Support: Extend voice features to family members and caregivers
  • Multilingual Support: Voice experiences in multiple languages
  • Enterprise Readiness: Full HIPAA compliance and security audits for clinical deployment
  • A2A Protocol Integration: Agent-to-agent communication endpoints are exposed across services; future work will fully integrate A2A for seamless cross-service agent orchestration and inter-agent task delegation

🌟 Platform Evolution:

  • Patient Communities: Connect patients with similar diagnoses and treatment phases
  • Outcome Tracking: Long-term survivorship support and monitoring

📚 Research Directions:

  • Pilot Study: Partner with clinical institutions to evaluate Oncydra's impact on symptom reporting and early toxicity detection
  • Academic Publication: Explore research questions around voice-based longitudinal PROs and miscommunication detection in oncology consultations

Bonuses

  • Medium article: "Building a Voice-First Cancer Care Companion with Google ADK and ElevenLabs", here
  • Prototyped initial app in Google AI Studio
  • Developed in Google Antigravity IDE

🫶 26 agents, 7 services, 5 voice personas, but at its core, Oncydra is just one thing: a companion that listens when patients need it most.

Built With

Share this project:

Updates