Elevate MediLens

AI-Powered Medication Label Assistant with Advanced OCR, FDA lookups, speech synthesis, and personalized health alerts

Comment 2

🧠 About the Project Elevate MediLens is an AI-driven web application designed to empower patients and reduce medication errors by transforming complex medication labels into clear, actionable, and personalized health information. Built using Google's Gemini API, the openFDA API, Google Cloud's Text-to-Speech, and YouTube Data API v3 for relevant medication-related videos. This web app provides instant analysis, data verification, and voice feedback to make medication safety accessible to everyone.

🎯 Inspiration Elevate MediLens was inspired by a critical issue: over 1.3 million people are harmed annually by medication errors in the U.S., leading to $42 billion in losses and 7,000–9,000 deaths. A major cause is the misinterpretation of complex medication labels, especially among the elderly, visually impaired, and medically unaware. This safety gap runs counter to the UN’s Sustainable Development Goal 3 (Good Health and Well-being). Our aim was to build a reliable, on-demand assistant that bridges the gap between medical jargon and patient understanding, making medication use safer for everyone.

🔧 How We Built Elevate MediLens Elevate MediLens is an AI-driven web application that transforms a simple photo of a medication label into clear, actionable, and personalized health information. Our workflow is built on a stack of powerful, integrated technologies.

📸 AI-Powered Label Analysis

  1. A user captures the medication label live scanning by using their device's camera or by uploading an image.
  2. Google's Gemini 2.5 Flash API instantly performs Optical Character Recognition (OCR) to read the label.
  3. Beyond just reading, the AI parses and structures the critical data: brand name, generic name, strength, dosage instructions, and warnings.

🛡️ FDA Verification for Accuracy

  1. To ensure reliability, the extracted drug name is cross-referenced with the openFDA API.
  2. This step validates the medication and enriches the results with official information, providing an essential layer of trust and safety.

🔊 Accessibility & Education for All

  1. Multi-Language Support: Integrated translation capabilities break down language barriers, making the information accessible to a global audience.
  2. Text-to-Speech (TTS): For visually impaired users, Google Cloud's TTS API reads the information aloud in a clear, natural-sounding voice.
  3. Educational Videos: The YouTube API fetches relevant videos about the medication, further enhancing patient understanding.

🔔 Personalized Health Alerts

  1. Users can create a simple health profile, listing known allergies and current medications.
  2. The system intelligently cross-references the scanned medication with this profile to generate personalized alerts for potential drug interactions or allergic reactions via coss verification from OpenFDA and Gemini 2.5 Flash API.

📄 Intuitive Reporting & Sharing

  1. The front-end, built with HTML, CSS, and JavaScript, features a clean dashboard with a "Quick Facts" tab and a "Full Summary" tab.
  2. A robust reporting feature allows users to download the analysis as a JSON or PDF file, or share it directly via email with caregivers and healthcare providers.

🧪 Field Testing & Support We tested Elevate MediLens across diverse real-world conditions—blurry images, angled shots, low lighting, small fonts, and complex label designs to ensure robustness and accuracy.

💡 What I Learned

  1. Gained hands-on experience integrating multiple AI and data APIs like Gemini, openFDA, YouTube, and TTS.
  2. Sharpened front-end development skills while focusing on clean design and accessibility.
  3. Learned how to build a system that delivers reliable outputs even with low-quality or imperfect inputs.
  4. Deepened my understanding of patient safety, health communication, and the real-world impact of tech.
  5. Learned how to turn a complex, high-stakes problem into a simple, usable solution that can scale.

🧱 Challenges Faced

  1. YouTube API Noise: Initial video results included irrelevant supplement content. Solved by refining queries using the drug’s generic name + clinical terms like "mechanism" and "side effects."
  2. Image Transparency Issues: Early logo files had solid backgrounds that broke UI aesthetics. Replaced with proper .png files with true transparency.
  3. CamanJS DOM Conflicts: Dynamic image loading caused CORS errors and DOM manipulation failures. CamanJS struggled with security restrictions on user-uploaded images.
  4. TTS Payload Complexity: Gemini’s TTS endpoint failed for our needs. Migrated to Google Cloud’s text: synthesize API, which required restructuring the payload (input, voice, audioConfig).
  5. Translation Bugs: Mixed-language output and unwanted prefixes broke readability. Some sections remained untranslated, requiring logic fixes.
  6. Label Data Confusion: The AI misread structured data. Needed strict formatting corrections.
  7. Broken PDFs: html2pdf.js rendered non-selectable image-based PDFs with layout issues and blank pages due to slow DOM rendering. Final Fix: Switched to pdfmake, allowing full programmatic PDF generation with real text, consistent structure, and stable rendering regardless of CSS complexity.

🏆 Accomplishments We're Proud Of

  1. End-to-End Patient Safety Assistant: Built more than an OCR tool—created a full-stack system combining scanning, AI analysis, FDA validation, alerts, and educational resources for a complete medication safety experience.
  2. Structured Data Extraction: Designed logic to extract clean, structured JSON from scanned labels—enabling consistent, accurate, and application-ready outputs critical for safety.
  3. Accessibility First: Integrated multi-language translation and TTS support to ensure usability for visually impaired users and non-native speakers—staying true to our mission of inclusivity.
  4. Personalized Health Alerts: Added a simple health profile system that generates real-time warnings about allergies or drug interactions—moving from passive info to proactive safety.
  5. Multi-API Integration: Seamlessly connected Gemini, openFDA, Google TTS, and YouTube APIs with robust error handling and workflow orchestration—delivering smooth functionality despite backend complexity.

🚀 Next Steps

  1. Native Mobile Apps
  2. HIPAA‑Compliant Backend
  3. Pharmacy & EMR Integration
  4. Expanded Regional Support
  5. Advanced Analytics Dashboard

Built With

Share this project:

Updates