MedLens

Empowering the visually impaired with AI in Healthcare: Self-identify Medications, Personalized Descriptions, and Easy Drug Management in One System!

Comment

“So I just guessed and decided to take three tablets every morning and two every evening”, said Mr. J, who has a visual impairment and has recently been discharged after in-hospital treatment for exacerbation of his chronic obstructive pulmonary disease, caused by difficulty reading descriptions on the medication bottle.

Inspiration 💡

The World Health Organization (WHO) reports that about 2.2 billion people worldwide have vision impairments, putting them at higher risk for medication errors. Vision impairment impacts reading medication labels, recognizing medications by color and shape, and reading markings on devices. Those who can't do this must rely on memory, strategies, or assistance from others.

In response to this pressing need, we developed MedLens — an innovative AI-powered medication management system — to help the visually impaired identify medications, give voice guidance, and provide personalized medication descriptions!

What it does 📓

  1. Smart Medication Identification: Employ cutting-edge Object Tracking and Optical Character Recognition (OCR) technology to automatically detect and identify medications and accurately read labels.

  2. Personalized Medication Summaries: Create personalized medication summaries by integrating patient-specific information, ensuring a tailored understanding of each drug's purpose and usage.

  3. Enhanced Accessibility for the Visually Impaired: Tailor-made for visually-impaired individuals, our system employs advanced speech generation technology to provide spoken guidance and readings, empowering users with independent and informed medication management.

How we built it 💻

AI-powered Technology:

  • Employ CloudVision Object Tracking API to precisely track medication bottles and packages while removing noise from the background
  • Leverage CloudVision OCR API to identify and scan labels and descriptions on medication
  • Utilize ChatGPT API for summarizing prescription data and providing personalized suggestions
  • Use Cloud Text-to-Speech API to provide voice guidance for the visually impaired

Web Development:

  • Harness Flask, HTML, JavaScript, and CSS for web application development.
  • Employ local storage to retain user information for future use.
  • Leverage Socket.io for facilitating communication between the Javascript frontend and Python backend.
  • Optimize performance with asyncio and ThreadPoolExecutor, enabling multithreading for faster processing and enhanced user experience.
  • All image resources are produced by Midjourney.

Challenges we ran into ⚓

  • All team members have more experience with the backend, so we need to learn the frontend from scratch.
  • Difficulty of deployment—The Python code of the Flask framework cannot be directly interpreted by web browsers. We tried different means and ended up with Google Cloud, which solves the problem.
  • Since our system comprises numerous components, optimizing processing speed and enhancing user experience became imperative. Eventually, we implemented multithreaded parallel processing to expedite the application.
  • Lack of sleep.

Accomplishments that we're proud of 👑

  • The OCR accurately and efficiently extracts information on a moving, arc-shaped object.
  • ChatGPT gives useful information about the meds.
  • The outcome is actually impactful. We believe the project can empower the visually impaired in healthcare and contribute to a more accessible world.

What we learned 🌈

  • The Flask framework which integrates Python into Web Development.
  • Google Cloud as the backbone of our project which offers access to various APIs and enables the deployment of the application.
  • Multithread processing
  • Never code 36 hours without sleep.

What's next for MedLens 🚀

  • Customized Models: Given our time constraints, developing customized models is currently unfeasible. We plan to pursue customized model training in the future.
  • Database: Add a database to store users’ information to create a more personalized assistant.
  • Language Model: Use a language model more performant in the medical field to provide more reliable consultation, like Google’s Med PaLM.
  • Medication Management: Allow users to effortlessly log medication details, including names, descriptions, and expiration dates, using OCR. This feature streamlines medication organization and tracking for users' convenience.
  • Smart Phone Application: Develop it as a smartphone app
  • Utilize a translation neural network to facilitate communication in various languages, assisting users in obtaining instructions for medications labeled in languages unfamiliar to them.
  • Design: Add more accessible UI

Built With

Share this project:

Updates