Dr. Listener

high-level design
Final Console
The Problem

Inspiration

12 million Americans experience outpatient diagnostic errors annually. For older adults with hearing loss or limited health literacy, the risk of leaving a doctor's appointment with unanswered questions or misunderstood instructions is especially concerning.

Studies suggest patients forget 40–80% of medical information soon after receiving it.

AI medical scribes capture what was said. But who catches what was never asked, never answered, or never understood?

That's why we built Dr. Listener — because being heard shouldn't mean being understood only halfway.

What it does

Dr. Listener is an accessibility-first, multi-agent AI companion that identifies four critical communication gaps in doctor–patient consultations:

Missed questions: Detects overlooked clinical questions using a 10-item AHA/ACC guideline-referenced chest-pain checklist.
Unanswered concerns: Flags patient worries that were never addressed.
Chart contradictions: Identifies inconsistencies between patient statements and medical records.
Comprehension breakdowns: Catches potential misunderstandings, including instructions that were never confirmed through teach-back.

Unlike traditional AI scribes, Dr. Listener doesn't just summarize conversations — it checks what the conversation missed.

Designed to minimize alert fatigue, it uses subtle in-visit indicators and consolidates unresolved concerns at wrap-up. The patient-facing experience focuses on accessible captions, read-aloud support, and plain-language care summaries.

How we built it

We built a deterministic, multi-agent architecture using:

Deepgram: Live speech-to-text with speaker diarization.
Python + Streamlit: Consultation interface and turn-by-turn orchestration.
Groq + LLMs: Specialized agents for clinical checklist coverage, concerns, contradictions, and comprehension.
Pydantic: Structured, validated AI outputs.
Presidio + spaCy: Best-effort de-identification before downstream LLM processing.
Evidence verification: Checks candidate flags against transcript quotes, medical-chart entries, and guideline-referenced checklists.

We also developed synthetic clinical conversations with deliberately planted gaps to evaluate the system's effectiveness.

Challenges we ran into

Speaker diarization: Distinguishing doctor and patient voices accurately during fast, overlapping conversations.
Detecting the unsaid: Identifying questions that were never asked, rather than merely summarizing existing dialogue.
False positives: Differentiating genuine misunderstandings from ordinary conversational responses.
Alert fatigue: Designing an assistant that supports clinicians without interrupting sensitive consultations.

Accomplishments that we're proud of

Built a multi-agent engine targeting four distinct consultation gap types.
Developed an evidence-verification layer instead of trusting unverified AI predictions.
Integrated live speaker-labeled transcription with structured clinical analysis.
Designed an accessibility-first workflow for patients with hearing loss and low health literacy.
Created reproducible synthetic evaluation cases with known gaps and clean controls.

Our biggest innovation: moving healthcare AI from passive documentation to proactive communication-gap detection.

What we learned

Accurate transcription doesn't guarantee effective communication.

We learned that identifying missing information requires structured clinical expectations, reliable speaker attribution, and evidence-backed reasoning.

Most importantly, accessibility cannot be an afterthought. It must shape how healthcare AI listens, verifies, and communicates.

What's next for Dr. Listener

We're working toward:

≥80% precision and recall in detecting communication gaps.
≤1 false flag per visit to minimize alert fatigue.
Evidence-backed flags with traceable transcript or chart references.
More accurate speaker diarization and multilingual accessibility.
Clinician-reviewed checklists, retrieval-grounded patient education, and secure EHR integration.

These are development goals, not validated clinical performance claims. Our current prototype uses synthetic data and is not a diagnostic tool.

Our vision: Healthcare where no critical question goes unasked, no concern goes unheard, and no patient leaves without understanding their care.