Inspiration

Medication errors from bad handwriting are a well-known, preventable harm. Mix-ups like Celebrex vs. Celexa, QD read as QID, or .5 mg read as 5 mg hurt patients every year. We wanted something a patient or pharmacist could point a phone at to get a second opinion before a misreading causes harm.

What it does

Script Doctor is an Android app. You photograph a handwritten prescription and it:

  • Transcribes each medication: drug, strength, form, route, frequency, duration and quantity.
  • Explains the directions in plain language.
  • Rates its confidence for each medication and lists other plausible readings.
  • Runs safety checks on the device: ISMP error-prone abbreviations, trailing or missing leading zeros, look-alike/sound-alike drugs, high-alert drugs read with low confidence, and missing strength or frequency.
  • Sums it all up as Low risk, Verify before use, or High risk: confirm with prescriber.

How we built it

  • Kotlin Multiplatform and Compose Multiplatform. Nearly all the code is shared in commonMain.
  • Images are downscaled to 1568 px or less with EXIF-aware rotation, then sent to Gemini's vision model using a strict JSON schema for the output and high thinking.
  • The safety rules are deterministic code, not AI, so the main warnings don't rely on the model.
  • History is stored only on the device. Requests are sent with store: false.

Challenges we ran into

  • Getting the model to admit uncertainty instead of guessing with confidence. We fixed it with per-medication confidence scores and a list of alternative readings.
  • Calling the API over plain HTTP with Ktor, because no platform SDK works in shared code.
  • Cutting scope mid-hackathon. We dropped iOS to focus on Android and switched model providers part way through.

Accomplishments that we're proud of

  • Safety checks that still catch dangerous patterns when the AI reads the script correctly.
  • An honest UI that shows uncertainty instead of hiding it.
  • A working, installable APK, with tests for the safety rules and the API client.

What we learned

  • In healthcare AI, showing how confident the model is matters as much as being accurate.
  • Simple fixed rules and an LLM work well together: the model reads the handwriting, the rules catch the known dangers.
  • Structured outputs make an LLM's answer reliable enough for an app to depend on.

What's next for Script Doctor

  • Route requests through a backend so no API key is stored on the phone.
  • Check against a real drug database such as RxNorm for interactions and dose ranges.
  • Bring back iOS.
  • Test with pharmacists on real, consented prescriptions.
  • Support more languages for the plain-language directions.

Built With

  • accessibility
  • ai-safety
  • android
  • compose-multiplatform
  • gemini
  • gemini-api
  • handwriting-recognition
  • healthcare
  • ismp
  • jetpack-compose
  • kotlin
  • kotlin-multiplatform
  • ktor
  • llm
  • medical-errors
  • medication-safety
  • ocr
  • patient-safety
  • prescription
  • structured-outputs
Share this project:

Updates

Submission history