RxPilot

Inspiration

I'm a pharmacist in Egypt, and handwritten prescriptions are still everyday reality at my counter. On a busy evening — patients waiting, phones ringing — a dangerous combination can slip through: someone on warfarin handed a new prescription for an anti-inflammatory, in handwriting nobody can fully read. Medication errors like these hurt real people, everywhere.

Most "AI for healthcare" demos get one thing badly wrong. They either remove the pharmacist entirely — dangerous and illegal — or wrap a chatbot around a prompt and call it clinical software. I wanted the opposite: automate the whole tedious workflow, but make the human checkpoint impossible to skip. Like an autopilot in a cockpit. It flies the route; the captain always lands the plane.

What it does

RxPilot takes a photo of a handwritten prescription (Arabic or English) and runs the full pharmacy workflow end to end:

• Reads the handwriting with Qwen3-VL and extracts every medication into structured data — with a confidence score per reading.
• Normalizes drug names — brand names, misspellings, local market brands — to the generic ingredient via RxNorm, with a Qwen fallback for regional brands.
• Runs a three-layer safety engine: a curated, clinically reviewed table of dangerous interactions (deterministic — caught every time); an allergy check against the patient's stored profile; and an AI clinical review for doses, duplication, and red flags.
• Stops at a decision gate: BLOCKED, NEEDS_REVIEW, or CLEARED. Nothing is dispensed until the pharmacist acts. If they need to change something, they edit a structured final list — that list becomes the single source of truth downstream.
• Generates outputs: deterministic dispensing labels (a label is a legal document — no generative model touches it) and a patient counseling sheet written by Qwen in Arabic or English, printed on a single A4 page.
• Remembers: approved medications go into the patient's profile, so next month's prescription is checked against this month's. Every step is timestamped in an audit trail.

How I built it

The AI layer runs entirely on Alibaba Cloud Model Studio (DashScope) — Alibaba Cloud's managed large-model platform. Every prescription makes live inference calls to Alibaba Cloud: Qwen3-VL reads the handwriting, and Qwen text models handle clinical reasoning, brand-name resolution, and bilingual counseling. The backend is Node.js and Express (serverless-ready for Alibaba Cloud Function Compute), connecting to Model Studio through the OpenAI-compatible endpoint. Drug normalization uses the RxNorm API from the U.S. National Library of Medicine.

The frontend is a single-file bilingual dashboard (English and Arabic, full RTL) served by the backend. Patient profiles and cases live in a JSON store behind a clean interface, designed to swap to Alibaba Cloud Tablestore without touching the rest of the code.

The design principle running through all of it: deterministic where safety demands it, generative where it genuinely helps. Interaction rules and labels are code. Handwriting, brand fallbacks, and patient counseling are AI. Mixing those up is how people get hurt.

Challenges I ran into

The hardest problem wasn't reading handwriting — Qwen3-VL handled that better than I expected. It was trust. How do you let a model touch a prescription without letting it make a decision it shouldn't? The answer became confidence thresholds and a strict split between deterministic and generative layers. Anything below 0.8 confidence, any unresolved drug name, any major interaction — the case cannot auto-clear.

Free-text modifications were another trap. Letting a pharmacist type "swap the painkiller" looks convenient until a misread note prints the wrong label. Replacing that with a structured editable list cost three clicks and removed an entire class of failure.

Accomplishments that I'm proud of

During testing, the AI review layer flagged a warfarin–omeprazole interaction that wasn't in my curated table yet — a real, correct clinical catch. That was the moment the two-layer design proved itself: the deterministic layer guarantees the known killers, the AI layer hunts for what I didn't think to write down.

I'm also proud that the whole thing is genuinely bilingual — a pharmacist in Egypt can work in Arabic, and a patient gets counseling in the language they actually read.

What I learned

Prompt a model like you'd brief a junior colleague — my biggest quality jumps came from telling Qwen exactly what not to do: never invent a dose, never add a medication, lower your confidence when unsure. And for the Middle East specifically, having multimodal reading and multilingual writing in one model family is the whole game.

What's next for RxPilot

Real pilot testing in a working pharmacy, a larger clinically reviewed interaction set, and integration with pharmacy dispensing systems. The destination isn't a hackathon submission — it's the counter of every busy pharmacy where a tired human is the last line of defense. That human deserves a copilot. Preferably one that knows it isn't the captain.

Built With

• alibaba-cloud
• css
• express.js
• healthcare
• html
• javascript
• model-studio
• node.js
• qwen
• qwen3-vl
• rxnorm