ClarifyRx – Plain-Language Medical Label Translator

Millions misuse medication because labels are too confusing to understand. ClarifyRx turns a photo of any prescription into plain-language instructions and critical warnings - Multilingual.

Comment

🩺 ClarifyRx — Plain-Language Medical Instruction Translator

Tagline: Empowering patients through plain-language medical literacy — upload any prescription label or discharge note and get instant, simplified explanations in English, Hindi, or Marathi.

💡 Inspiration

Nearly everyone has stared at a complex prescription label or hospital discharge note and understood maybe half of it. For most, it's a minor inconvenience — we search online or call the pharmacy. But for elderly patients, low-literacy individuals, or non-native speakers, medical jargon isn't just confusing — it's a critical safety hazard.

Missed warnings, incorrect dosages, and dangerous drug interactions caused by misunderstood medical instructions are documented leading drivers of medication errors and preventable emergency room readmissions.

We built ClarifyRx to close this health literacy gap with zero friction:

  • ❌ No app downloads
  • ❌ No user accounts or login walls
  • ❌ No complicated text forms
  • ⚡ Just a photo, and instant, crystal-clear answers.

✨ What It Does

ClarifyRx lets anyone snap or upload a photo of a prescription label, medicine bottle, or discharge instruction sheet. It extracts the raw medical text directly from the image and translates it into plain, simple language at a 5th-grade reading level in English, Hindi, or Marathi.

Key Features:

  • 📸 Direct Multimodal Image Translation: No manual typing required — drag-and-drop or snap a photo of any pill bottle or medical document.
  • 🗣️ Built-in Text-to-Speech (Audio Reader): One-click voice output so low-literacy or visually impaired users can listen to their medication instructions read aloud in their native language.
  • ⚠️ High-Contrast "Watch Out For" Safety Alerts: Critical drug interactions, timing rules, and "do not consume with..." warnings are extracted into prominent, red-accented warning cards so danger notes are never buried in paragraph text.
  • 📊 Instant Reading-Level Analysis: Computes real-time Flesch-Kincaid Grade Level metrics comparing the original complex medical text (e.g., Grade 14 - College Level) against the rewritten output (Grade 5 - Plain Language).
  • 💬 Interactive Health Literacy Assistant: A context-aware follow-up Q&A chat box where users can ask questions like "Should I take this with food?" or "What if I miss a dose?" with built-in medical guardrails that automatically advise consulting a doctor or pharmacist when appropriate.
  • 🧪 Pre-Loaded Interactive Demos: Instant single-click sample loaders for Blood Thinners (Warfarin), Antibiotics (Amoxicillin), and Post-Surgery Discharge notes for instant judge evaluation.

🛠️ How We Built It

ClarifyRx is engineered as a high-performance single-page web app built with React, TypeScript, and Tailwind CSS, leveraging Google's Gemini Multimodal REST API:

  • Multimodal Single-Pass Pipeline: Using Gemini's vision-language multimodal capabilities, we send raw base64 image data directly alongside system prompt instructions. Gemini reads the image text (OCR) and rewrites it into plain language in a single REST request, returning structured JSON.
  • Strict Structured JSON Schema: We enforced strict JSON schemas (extracted_text, summary, warnings) with robust client-side markdown fence stripping and try/catch fallbacks to ensure reliable UI rendering.
  • Client-Side Flesch-Kincaid Reading Score Engine: We built a zero-latency client-side readability calculator that parses word, sentence, and phonetic syllable counts in real-time, providing immediate visual feedback without extra API overhead.
  • Accessible, Human-Centered Healthcare Design: Crafted with 16px+ legible typography, soothing healthcare palette (soft blues and clean whites), ARIA accessibility standards, mobile-first responsiveness, and Web Speech API audio synthesis.

🚧 Challenges We Ran Into

  1. Ensuring JSON Rigor from Generative Output: Free-form AI responses frequently break structured UIs. We solved this by constraining the prompt to output strict JSON schemas and building resilient client-side parsers that strip code fences (`json) automatically.
  2. Preserving Safety Criticality Across Languages: Translating medical warnings into Hindi or Marathi risks softening critical instructions. We fine-tuned system prompts to preserve urgent imperative phrasing for safety warnings.
  3. Designing for Patient Trust & Guardrails: In health tech, hallucination is unacceptable. We explicitly instructed the engine to stick strictly to source text facts and defer to licensed pharmacists or physicians whenever questions exceed the label's explicit scope.

🎉 Accomplishments That We're Proud Of

  • ⚡ End-to-End Multimodal Production Flow: Built a fully functional, zero-latency image-to-translated-speech pipeline.
  • 🌐 Inclusive Multi-Language Access: Prioritized Hindi and Marathi alongside English — bringing health literacy to millions of underserved non-English native speakers.
  • 🔊 Implemented Native Audio Accessibility: Integrated browser-native Speech Synthesis so the app works for individuals who cannot read at all.
  • 🛡️ Safety-First UX: Made warning flags impossible to miss, treating safety alerts as primary UI components rather than footnotes.
  • 🎯 Zero-Friction Access Model: Built a privacy-preserving web app with no database footprint and no registration barriers.

🧠 What We Learned

We learned that building AI for healthcare is fundamentally an exercise in constraint, clarity, and restraint. The hardest challenge wasn't getting AI to summarize text — it was getting it to summarize responsibly: never inventing advice, keeping warnings prominent, and knowing when to tell a patient to call a professional. We also validated how removing typing friction via vision multimodal input completely changes accessibility for elderly users.

🚀 What's Next for ClarifyRx

  • 🌐 Expand Regional & Global Languages: Add Spanish, Bengali, Tamil, Telugu, and Arabic support.
  • 📱 PWA & Offline Micro-Model Support: Package ClarifyRx as a Progressive Web App (PWA) with lightweight local vision models for rural or low-connectivity clinics.
  • 💊 Medication Schedule Reminders & QR Exports: Allow users to save simplified instructions directly to their mobile calendar or export printable medication wallet cards.
  • 🏥 Clinical Pilot Program: Partner with local community health centers, pharmacies, and elder-care facilities to collect real-world patient feedback.

Built With

Share this project:

Updates

Submission history