Medical Scribe

AI-powered clinical documentation that turns doctor-patient conversations into structured medical reports — saving physicians 10+ hours of paperwork every week.

Comment

Inspiration

Doctors in Vietnam and across Southeast Asia spend 10–15 hours every week writing clinical notes — time stolen from patients. We saw this firsthand: physicians typing furiously during consultations, trying to remember details afterward, or dictating notes late at night. The paperwork is not medicine. It's burnout.

We built Medical Scribe to give that time back.

What it does

Medical Scribe turns a doctor-patient conversation into a structured clinical report in under 2 minutes.

The workflow:

  1. Record or upload the consultation audio (Vietnamese or English)
  2. Whisper transcribes the conversation to text
  3. Qwen AI extracts the clinical information — chief complaint, symptoms, vital signs, diagnoses with ICD-10 codes, medications with dosage
  4. A structured report is generated with confidence scores for each field
  5. The doctor reviews and approves — no typing required

The result: a complete, accurate clinical note with ICD-10 codes, medication instructions, and a clinical impression — ready in the time it takes to walk to the next patient.

How we built it

Backend: FastAPI (Python) with SQLAlchemy ORM and SQLite
AI — Transcription: OpenAI Whisper API (whisper-1) for Vietnamese/English audio
AI — Extraction: Qwen (qwen/qwen-plus) via OpenRouter for structured clinical data extraction
Frontend: React with a consultation dashboard, audio upload, and report viewer
Prompting: Few-shot prompting in both Vietnamese and English, with strict JSON output format, ICD-10 codes, and per-field confidence scores (0.0–1.0)

Qwen is the core of the system. We use it to extract:

  • Chief complaint
  • Symptom list
  • Vital signs
  • Physical findings
  • Diagnoses (with ICD-10 codes and confidence)
  • Medications (name, dosage, frequency)
  • Clinical impression

We kept Qwen temperature at 0.3 — low enough for clinical consistency, so the same transcript always produces the same extraction.

Challenges we ran into

  • Vietnamese medical language is a mix of formal Vietnamese, French loan words, and English medical terms. Getting Qwen to handle this correctly required careful few-shot examples in both languages.
  • Audio transcription accuracy on noisy real-world recordings. We fell back to Whisper's auto-detect mode for mixed-language consultations.
  • Structured output reliability — medical data must be exact, not approximate. We implemented retry logic with exponential backoff and JSON validation to ensure clean extraction every time.
  • Trust and confidence — doctors won't use AI output they can't verify. We added per-field confidence scores so physicians know exactly how certain the system is about each piece of extracted data.

Accomplishments that we're proud of

  • End-to-end pipeline working: audio → transcript → Qwen extraction → structured report, in under 2 minutes
  • Bilingual support: Vietnamese and English consultations both produce accurate clinical reports
  • ICD-10 codes extracted automatically from natural conversation
  • Confidence scores on every field — making AI output trustworthy for clinical use
  • Demo mode: the full pipeline runs without any API keys, making it easy for judges to evaluate

What we learned

  • Qwen handles medical Vietnamese surprisingly well with few-shot prompting — better than we expected
  • Low temperature (0.3) is critical for clinical consistency — higher temperatures produce creative but medically unreliable outputs
  • Doctors trust AI more when they can see why it made a decision — confidence scores matter more than accuracy alone
  • The hardest part of healthcare AI is not the model — it's earning the trust of the physician reviewing the output

What's next for Medical Scribe

  • Real-time transcription during the consultation — live note-taking as the doctor speaks
  • EHR integration — push reports directly to hospital systems (HL7 FHIR format)
  • Specialist templates — different extraction schemas for cardiology, pediatrics, oncology
  • Doctor correction loop — when a doctor edits a report, that correction feeds back to improve future extractions
  • Deployment on Alibaba Cloud ECS for production use at Vietnamese clinics

Built With

Share this project:

Updates

Submission history