The Active Clinical Guardian

Inspiration

Real-time clinical safety is a life-or-death problem. Every year, preventable adverse drug events harm millions of patients — often because critical drug interaction warnings arrive too late, buried in static alerts that doctors learn to ignore. We were inspired to build an active guardian that listens alongside the doctor and intervenes in the moment, using voice AI to literally interrupt a dangerous prescription before it's written.

What it does

The Active Clinical Guardian is a real-time clinical safety monitoring system that:

• Listens to doctor-patient consultations via ElevenLabs Scribe v2 real-time transcription
• Understands clinical intent using Dedalus Labs structured output extraction (medications, procedures, diagnoses)
• Reasons about drug safety using K2 Think's System 2 reasoning engine, cross-referenced with patient records from Snowflake and RAG-retrieved clinical guidelines
• Interrupts the doctor with a voice alert (ElevenLabs TTS) when a dangerous drug interaction or allergy conflict is detected — in real time
• Documents the encounter automatically with AI-generated SOAP notes and CPT/ICD-10 billing codes via Flowglad

How we built it

• Backend: FastAPI (Python) orchestrating a pipeline: Dedalus → Snowflake Cortex RAG → K2 Think → ElevenLabs
• Frontend: Next.js + React with real-time WebSocket communication, TailwindCSS + shadcn/ui
• Transcription: ElevenLabs Scribe v2 Realtime (streaming PCM audio over WebSocket with VAD commit strategy)
• Clinical NLP: Dedalus Labs SDK with Pydantic structured outputs for 100% reliable JSON extraction of clinical intent
• Safety Reasoning: K2 Think v2 via OpenAI-compatible API for System 2 deep reasoning over drug interactions, allergies, and clinical guidelines
• Patient Data & RAG: Snowflake Cortex with EMBED_TEXT_768 embeddings and VECTOR_COSINE_SIMILARITY for semantic search over clinical guidelines
• Voice Interruptions: ElevenLabs TTS streaming (Turbo v2.5) for low-latency spoken safety alerts
• Billing: Flowglad API for automated CPT code generation and invoice creation

Challenges we ran into

• Audio format mismatch: Browsers send webm/opus but ElevenLabs Scribe needs raw PCM 16-bit 16kHz mono — we had to build a custom AudioWorklet pipeline in the frontend
• Connection flooding: ElevenLabs Scribe has a 15-second inactivity timeout, causing rapid reconnect loops. We solved this with lazy connection (open on first audio chunk) and a cooldown-gated reconnect strategy
• Duplicate transcripts: Partial and committed transcripts from Scribe were both being appended, causing duplicates. We implemented in-place replacement for partials with permanent addition only on commit
• Orchestration complexity: Coordinating five async services (Dedalus, Snowflake, K2, ElevenLabs STT, ElevenLabs TTS) per safety check required careful async pipeline design with graceful fallbacks

Accomplishments that we're proud of

• Sub-second safety detection: From the moment a doctor says a dangerous drug name to the voice interruption playing, the full pipeline (transcription → intent extraction → RAG → reasoning → TTS) completes in real time
• True voice interruption: The system doesn't just show a red banner — it speaks to the doctor, creating an unmissable alert that mirrors how a pharmacist would intervene
• Dedalus-orchestrated pipeline: Using Dedalus structured outputs to extract clinical intent before RAG search dramatically improves retrieval precision — we search for "sumatriptan sertraline SSRI interaction" instead of raw transcript text
• End-to-end automation: From live audio to SOAP note to billing invoice, the entire consultation workflow is handled

What we learned

• The importance of targeted RAG queries — raw transcript dumps into vector search produce noisy results; structured intent extraction first makes all the difference
• System 2 reasoning (K2 Think) catches interactions that simple rule-based lookups miss, especially for complex multi-drug scenarios
• Real-time audio streaming requires careful connection lifecycle management — lazy initialization, reconnect cooldowns, and per-session isolation
• Structured outputs (Dedalus/Pydantic) eliminate an entire class of parsing bugs in AI pipelines

What's next for The Active Clinical Guardian

• Multi-language support: Leverage ElevenLabs Scribe's multilingual capabilities for non-English consultations
• MCP tool integration: Connect Dedalus to external medical reference databases via Model Context Protocol
• EHR integration: Push SOAP notes and safety alerts directly into Epic/Cerner via FHIR APIs
• Continuous learning: Feed safety check outcomes back into Snowflake to improve RAG retrieval over time
• Multi-provider support: Scale to support multiple concurrent consultations with provider-specific safety profiles

Built With

• dedalus
• elevenlabs
• fastapi
• flowglad
• k2-think
• next.js
• python
• snowflake