DocTalk

Live Console
Landing Page
DocTalk
System Design Architecture
Problem

Inspiration

12 million Americans experience outpatient diagnostic errors every year. For older adults with hearing loss or limited health literacy, the risk of leaving an appointment with unanswered questions or misunderstood instructions is even greater. Studies suggest patients forget 40–80% of medical information soon after receiving it.

AI medical scribes capture what was said. But who catches what was never asked, never answered, or never understood?

That's why we built DocTalk: because being heard shouldn't mean being understood only halfway.

What it does

DocTalk is an accessibility-first, multi-agent AI companion that identifies four critical communication gaps in doctor–patient consultations:

Missed questions: detects overlooked clinical questions using a 10-item, AHA/ACC guideline-referenced chest-pain checklist.
Unanswered concerns: flags patient worries that were never addressed.
Chart contradictions: identifies inconsistencies between what the patient says and the medical record.
Comprehension breakdowns: catches potential misunderstandings, including instructions never confirmed through teach-back.

Unlike traditional AI scribes, DocTalk doesn't just summarize the conversation. It checks what the conversation missed.

Beyond cardiology: DocTalk is built on pluggable condition modules. Alongside Cardiology, we've extended it to Oncology and Pediatric Oncology, where a missed question or misunderstood plan matters most. The pediatric module also includes the child's voice and checks that parents understood the plan.

Designed to minimize alert fatigue, DocTalk uses subtle in-visit indicators and consolidates unresolved concerns at wrap-up. The patient-facing experience offers accessible captions, read-aloud support, and plain-language care summaries.

How we built it

A deterministic, multi-agent architecture:

Deepgram: live speech-to-text with speaker diarization
Python + Streamlit: consultation interface and turn-by-turn orchestration
Groq + LLMs: specialized agents for checklist coverage, concerns, contradictions, and comprehension
Pydantic: structured, validated AI outputs
Presidio + spaCy: best-effort de-identification before downstream LLM processing
Evidence verification: every candidate flag is checked against transcript quotes, chart entries, and guideline-referenced checklists
Pluggable modules: each specialty is a swappable pack (checklist, gap checks, glossary, patient profiles), so the same agents run across Cardiology, Oncology, and Pediatric Oncology

We also built synthetic clinical conversations with deliberately planted gaps to evaluate the system.

Challenges we ran into

Speaker diarization: telling doctor and patient apart in fast, overlapping conversation
Detecting the unsaid: finding questions that were never asked, rather than summarizing what was said
False positives: separating real misunderstandings from ordinary conversational replies
Alert fatigue: supporting clinicians without interrupting sensitive consultations

Accomplishments that we're proud of

A multi-agent engine targeting four distinct consultation gap types
An evidence-verification layer instead of trusting unverified AI predictions
Live speaker-labeled transcription combined with structured clinical analysis
An accessibility-first workflow for patients with hearing loss and low health literacy
A multi-specialty design that extends from Cardiology to Oncology and Pediatric Oncology
Reproducible synthetic evaluation cases with known gaps and clean controls

Our biggest innovation: moving healthcare AI from passive documentation to proactive communication-gap detection.

What we learned

Accurate transcription doesn't guarantee effective communication.

Finding missing information takes structured clinical expectations, reliable speaker attribution, and evidence-backed reasoning. Above all, accessibility can't be an afterthought. It has to shape how healthcare AI listens, verifies, and communicates.

What's next for DocTalk

≥80% precision and recall in detecting communication gaps
≤1 false flag per visit to minimize alert fatigue
Evidence-backed flags with traceable transcript or chart references
More accurate speaker diarization and multilingual accessibility
Expanding to more specialties, with clinician-reviewed checklists for each
Retrieval-grounded patient education and secure EHR integration

Note: These are development goals, not validated clinical performance claims. Our prototype uses synthetic data and is not a diagnostic tool.

Our vision: healthcare where no critical question goes unasked, no concern goes unheard, and no patient leaves without understanding their care.