EchoDoc - Voice Clinical Assistant

Ambient Voice Listener Client (live_voice_test.py) with Integrated Speech Recognition and TTS Synthesis
Ambient Voice Listener Client (live_voice_test.py) with Integrated Speech Recognition and TTS Synthesis

Inspiration## Inspiration

In high-pressure clinical environments, healthcare professionals spend hours manually dictating and typing patient notes. Under time pressure, critical polypharmacy risks, age-specific contraindications, or speech-to-text misspellings can lead to severe adverse drug events (ADEs). We built EchoDoc to serve as an ambient, voice-first clinical assistant that not only transcribes doctor dictation hands-free but also acts as a real-time safety copilot—evaluating drug interactions and age-specific warnings across ages 1 to 100 before prescriptions are finalized.

What it does

EchoDoc listens to natural clinician dictation, processes the voice transcript, and instantly evaluates potential medical risks. Key features include:

Hands-Free Ambient Voice Dictation: Listens for wake words (e.g., "Alexa") and processes complete patient history entries ending with "over".
Phonetic STT Defense Engine: Handles speech-to-text misinterpretations by mapping fuzzy phonetic dictations (e.g., "warren" for "warfarin") to actual pharmaceutical terms.
Full-Lifecycle Age Sensitivity (Ages 1–100): Dynamically extracts patient age from voice input and applies age-tailored safety alerts—such as flagging Aspirin in pediatric patients due to Reye's syndrome risk or highlighting fall risks with Benzodiazepines in elderly patients.
Critical Interaction Alerts: Detects severe drug-drug interactions (e.g., Sildenafil + Nitroglycerin causing fatal hypotension, Alprazolam + Opioids causing fatal respiratory depression, or Warfarin + NSAIDs causing internal hemorrhage) and provides real-time spoken audio alerts.

How we built it

Backend API: Built with FastAPI and Python for ultra-low latency request handling.
Speech-to-Text (STT): Utilized Python's SpeechRecognition library configured for ambient listening and wake-word triggers.
Safety Rules Engine: Developed custom regular expression (Regex) parsers and phonetic dictionary matchers for real-time age extraction and multi-category risk analysis.
Text-to-Speech (TTS): Integrated native Windows SAPI5 / AWS Polly speech synthesis for reliable, crisp spoken feedback to the clinician.

Challenges we faced

Audio Synthesis Reliability: Initial implementations with third-party audio loops caused silent output failures on Windows environments. We overcame this by transitioning to direct Windows SAPI5 / AWS Polly synthesis.
Phonetic Misrecognitions: STT engines frequently hear medical terms as common names or slang (e.g., "warren" instead of "warfarin"). We addressed this by building a dedicated phonetic alias dictionary within our backend rules engine.
Complex Polypharmacy Logic: Structuring safety checks that balance pediatric contraindications, adult drug-drug interactions, and geriatric risks without adding latency required optimizing our parsing algorithms.

What we learned

We learned how to design robust, fault-tolerant clinical NLP pipelines that bridge voice UI with medical decision support systems. We gained deep insights into speech-to-text edge cases, dynamic age parsing, and building asynchronous FastAPI microservices optimized for real-time feedback.

What's next for EchoDoc

AWS Cloud Deployment: Containerizing the FastAPI service using Docker and deploying it on AWS EC2 / AWS Lambda for scalable multi-clinic access.
EHR Integration: Connecting EchoDoc directly to electronic health record systems (e.g., FHIR API integration) for auto-populating patient charts.
Expanded Drug Database: Integrating live clinical APIs (like RxNorm or First Databank) for dynamic, real-time access to global pharmaceutical databases.