RxRead_AI

Inspiration

We are from Peshawar, Pakistan, where handwritten prescriptions in mixed Urdu and English are the everyday norm. We have all watched a family member receive a prescription, stare at it in confusion, and have to hand it to someone else — a pharmacist, a relative, anyone who could decipher the doctor's handwriting — just to find out what medicine they had been given. This isn't a rare inconvenience. Studies show nearly 60% of handwritten prescriptions are partially illegible, and in the US alone, illegible prescriptions contribute to over 7,000 deaths every year. For patients in rural Pakistan, the Middle East, and other regions where handwritten prescriptions remain standard, the risk is even greater — there often isn't a pharmacist available 24/7 to help. We built RxRead AI because this is a problem we have lived, not just observed.

What it does

RxRead AI lets a patient photograph any prescription — handwritten or printed — and instantly receive a clear, structured explanation of what it says. The app identifies the doctor's name, each medicine, its dosage, frequency, duration, and any special instructions, then displays everything in plain, non-medical language. Every result includes a confidence score so the patient knows how reliable the AI's reading is. If confidence is low, the app explicitly recommends verifying with a pharmacist rather than presenting an uncertain answer as fact. RxRead AI does not diagnose, prescribe, or modify what the doctor wrote — it only reads and clarifies.

How we built it

Frontend: React + TypeScript, with a mobile-first responsive upload flow, progress tracking, and token refresh handling via Axios interceptors Backend: Node.js + Express with MongoDB for data storage, JWT authentication, Zod for input validation, and Multer for handling prescription image uploads AI layer: Google Gemini Vision API reads the prescription image and extracts structured fields (medicine, dosage, frequency, duration, instructions). We added retry logic with exponential backoff and a confidence-scoring layer on top of the raw model output Security: Helmet security headers, CORS origin whitelisting, rate limiting, and server-side-only image processing — no prescription photos are shared with third parties

Challenges we ran into

Handwriting recognition for mixed Urdu/English medical shorthand was the hardest technical problem — standard OCR tools fail completely on this, which is exactly why we needed a vision-capable AI model rather than a rules-based approach. We also had to think carefully about responsible AI design rather than just accuracy. A wrong medicine name read with high confidence could be dangerous, so we built explicit confidence thresholds and "verify with a pharmacist" warnings directly into the output — not as an afterthought, but as a core part of the product. Coordinating a global team — based across Pakistan, Saudi Arabia, and the US — across time zones during a tight hackathon build window was also a real challenge in itself.

Accomplishments that we're proud of

We built and deployed a fully working end-to-end system — not just a mockup — that takes a real photo of a real handwritten prescription and returns an accurate, structured, plain-language result in under 30 seconds. We are especially proud that this app was inspired by and tested against prescriptions our own families have received.

What we learned

We learned how much responsible AI design matters in a healthcare-adjacent product — that showing confidence levels and clear disclaimers is not optional polish, it's a core safety feature. We also learned how to take an AI vision model from a single prompt experiment to a structured, production-style pipeline with validation, retries, and error handling.

What's next for RxRead_AI

Full multilingual output in Urdu, Pashto, and Arabic — not just English Prescription history and refill reminders Drug interaction warnings using public medicine databases Audio readout for patients with low literacy An offline-capable mobile app for areas with limited internet acces

Built With

• axios
• express.js
• google-gemini-vision-api
• helmet
• javascript
• mongodb
• mongoose
• multer
• netlify
• node.js
• react
• zod