Inspiration
Physicians spend nearly two hours completing electronic health record (EHR) paperwork for every single hour spent caring for patients. This administrative burden fuels record clinician burnout and detracts from the human connection in medicine.
While ambient AI scribes have emerged, existing solutions often operate as ungrounded "black boxes": they hallucinate clinical details, overlook dangerous drug-allergy interactions, and stop at note generation without closing the loop with the patient.
We built Escriba on the principle of "Autonomous Planning, Human Approval": an intelligent clinical co-pilot that listens, extracts, verifies clinical safety against contraindications, and keeps the clinician firmly in control while extending care directly to the patient's phone.
What it does
Escriba manages the complete clinical lifecycle across 5 end-to-end phases:
- Pre-Visit SMS Intake: Securely captures patient history, chief complaints, and symptoms before the consultation begins via automated Twilio SMS workflows.
- Real-Time Guardrails & Streaming Audio: Ingests live visit audio over WebSockets and runs real-time safety guardrails to detect contraindications (e.g., catching Bactrim prescribed to a sulfa-allergic patient or amlodipine allergy conflicts) before the patient leaves the clinic.
- Structured Clinical Synthesis: Uses OpenAI Structured Outputs to generate clean, professional SOAP notes, filtering out small talk while preserving social determinants of health (SDOH). Every proposed medication or order cites exact, verifiable transcript evidence.
- Formulary Check & 1-Click Clinician Approval: Cross-references proposed orders against formulary coverage, flags high-cost brand rejections (e.g., prior authorization for Ozempic) with recommended alternatives, and enables clinicians to review, edit, and lock notes with full audit trails.
- Closed-Loop Follow-Up & Triage: Translates complex discharge instructions into a 6th-grade reading level summary (with bilingual support, e.g., English clinical note + Spanish patient summary) sent directly via SMS, followed by an automated 48-hour check-in that flags complications for triage.
How we built it
- Core Backend: Built with Python 3.11 and FastAPI, orchestrating asynchronous API endpoints and full-duplex WebSockets for low-latency audio chunk streaming and guardrail alerts.
- Data Layer: SQLAlchemy 2.0 (async) with SQLite and PostgreSQL schemas modeling the clinical domain (
Patient,Encounter,ClinicalDocument, andIntakeChat). - AI & Clinical NLP: OpenAI Structured Outputs (
gpt-4o) backed by strict Pydantic models to eliminate hallucinations, enforce rigid clinical schemas, and bind proposed orders directly to transcript citations. - Safety Guardrail Engine: A dedicated non-blocking verification service that cross-references documented allergies against synthesized prescriptions and highlights drug interactions.
- Telephony & Messaging: Twilio SMS Webhooks hardened with HMAC-SHA1 signature verification for intake and outbound patient discharge instructions.
- Testing & Verification: A benchmark evaluation suite testing clinical safety, dialect variations, negation handling, and citation grounding across multi-case medical scenarios.
Challenges we ran into
- Eliminating Hallucinations & Ensuring Grounding: Medical documentation leaves zero room for confabulation. We designed strict validation schemas and bidirectional quote matching so that every order, medication, and dosage must point back to verbatim transcript evidence.
- Bilingual & Mixed-Language Consultations: Patients and clinicians often code-switch between English and Spanish during visits. We engineered our synthesis prompts to generate institutional-standard English SOAP notes while simultaneously delivering patient take-home summaries in the patient's spoken language.
- Low-Latency Safety Checks: Running LLM contraindication checks over continuous streaming audio without blocking WebSocket throughput required fine-tuning async backpressure and fail-visible alert pipelines.
Accomplishments that we're proud of
- 100% Benchmark Pass Rate: Validated against a rigorous clinical benchmark suite testing complex medical cases, dosage verifications, and contraindication detections.
- Fail-Visible Human-in-the-Loop Architecture: Designed an interface where AI suggests and verifies, but the clinician maintains absolute authority to approve, edit, and lock records before orders can be placed.
- Closing the Care Loop: Going beyond transcription to deliver actionable, readable health instructions directly into patients' hands via SMS with proactive complication monitoring.
What we learned
- Grounding Builds Trust: Clinicians don't just want an accurate note; they want instant verification. Providing clickable "View Source" jumps directly to transcript evidence creates trust that raw generation cannot.
- Strict Schemas are Essential in Healthcare AI: Free-form LLM outputs introduce unpredictable edge cases; leveraging OpenAI Structured Outputs with rigid Pydantic models is essential for clinical compliance and deterministic safety.
What's next for Escriba
- Autonomous Pre-Check-In Agent: Expand the initial SMS intake into a dynamic, multi-turn clinical intake agent that adaptively interviews patients before their visit, performs preliminary anamnesis, and flags acute symptoms directly to the triage queue.
- Agentic Post-Appointment Care & Smart Reminders: Deploy a dedicated post-visit follow-up agent that proactively engages patients via conversational chat—tracking medication adherence, sending scheduled dose and appointment reminders, answering recovery questions grounded in the doctor's plan, and escalating concerning symptoms to on-call staff.
- Direct EHR Integration: Connect via FHIR and HL7 APIs (Epic, Cerner, AthenaHealth) for seamless chart injection and direct CPOE (Computerized Provider Order Entry) staging.
- Multi-Speaker Diarization: Deploy robust ambient acoustic models capable of separating overlapping dialogue between doctor, patient, family members, and medical assistants.
- Real-Time Benefit Check (RTBC): Integrate live e-prescribing networks (like Surescripts) for real-time patient copay transparency and automated electronic prior authorizations (ePA).
Built With
- artificial-intelligence
- asr
- asyncpg
- audio-streaming
- fastapi
- gpt-4o
- healthcare
- healthtech
- llm
- machine-learning
- natural-language-processing
- openai
- postgresql
- pydantic
- pytest
- python
- rest-api
- sms
- speech-to-text
- sqlalchemy
- sqlite
- structured-outputs
- twilio
- uvicorn
- websockets