IntelliCare

Automated Medical Information Extraction and Patient Query Resolution Using OCR, NER, Fine-Tuned Models, and Retrieval-Augmented Generation (RAG)

Comment

Inspiration

Our inspiration came from the daily struggles patients face when trying to understand complex medical information in prescriptions and reports. Misunderstandings can lead to improper medication use and adverse health effects. We aimed to bridge this gap by creating a solution that makes medical information accessible and comprehensible for everyone.

What it does

Our project, IntelliCare, consists of two main components: the Prescription Scanner and the Report Scanner. The Prescription Scanner converts printed prescriptions into digital text, identifies and extracts medicine names, and provides detailed, easy-to-understand descriptions, including usage, dosage, side effects, and more. The Report Scanner digitizes medical reports, processes and summarizes the information, and delivers clear, insightful details about diagnoses, symptoms, and treatments in a patient-friendly format and a QA session on the patient report using RAG.

How we built it

We built IntelliCare using several advanced technologies. We employed OCR for text extraction, NER models for identifying medical terms, and fine-tuned our custom LLM on Mixtral-8x7B Instruct v1.0 for generating detailed medicine descriptions. For the Report Scanner, we used Sentence Transformers to embed text chunks and stored them in a vector database like Pinecone on AWS Cloud. Retrieval-Augmented Generation (RAG) was utilized to ensure accurate and relevant information retrieval.

Challenges we ran into

One of the main challenges was ensuring the accuracy of OCR technology in extracting text from handwritten prescriptions. Training the NER model to recognize a wide variety of medical terminologies was also a complex task. Additionally, fine-tuning the LLM to provide precise and relevant information required extensive data preprocessing and model optimization.

Accomplishments that we're proud of

We are proud to have created a solution that can significantly enhance patient understanding and safety by making medical information more accessible. Successfully integrating various advanced technologies to provide accurate and user-friendly information is a significant achievement. Our project not only simplifies complex medical jargon but also empowers patients to make informed decisions about their health.

What we learned

Throughout this project, we learned the importance of interdisciplinary collaboration in solving real-world problems. Combining expertise in machine learning, natural language processing, and healthcare allowed us to create a robust and effective solution. We also gained valuable insights into the challenges of medical data processing and the potential of AI in transforming healthcare.

What's next for IntelliCare

The next step for IntelliCare is to integrate online medicine purchasing with streamlined handling of both digital and handwritten prescriptions, making the process even more convenient for users. We also plan to extend our services to hospitals, offering features like seamless prescription processing, inventory management, and patient medication tracking. Our goal is to further enhance healthcare accessibility and efficiency, ensuring better outcomes for patients and healthcare providers alike.

Features

Prescription Scanner

  • Prescription Upload: Users can upload their prescription documents.
  • Text Extraction: Utilizes Keras-OCR with TensorFlow to accurately extract text from images.
  • Generative AI: Generates detailed query-answer like information about each extracted medicine.
  • Multi-Language Translation: Allows translation of medicine details into multiple languages.
  • AI Warning: Displays a caution message, advising users to consult their doctor for more insights.

RecordDelve 📄🔍

RecordDelve offers a comprehensive solution for managing and querying medical reports. It consists of two main divisions:

RecordScan

  • Medical Report Upload: Users can upload their medical reports.
  • Multi-Language Translation: Allows translation of medical report details into multiple languages.
  • Insight Extraction: Provides detailed insights into the medical report, helping users understand their health better.

RecordQuery

  • Report Storage: Uses vector databases to securely store medical reports.
  • Query Answering: Acts as a Retrieval-Augmented Generation (RAG) model to answer user queries based on their specific medical report, ensuring confidentiality and accuracy.

Revolutionizing Healthcare 🏥💻

IntelliCare is at the forefront of transforming the healthcare industry by integrating advanced AI technologies. Our platform offers:

  • Accurate Medication Information
  • Reliable Health Insights
  • Confidential Medical Record Handling

Join us in revolutionizing healthcare delivery and improving patient outcomes with IntelliCare. 🚀💊👨‍⚕️

Built With 🛠️

Our application leverages the following technologies:

Frontend & Backend:

  • Streamlit

Machine Learning & AI:

  • transformers
  • tensorflow==2.15.1
  • sentence-transformers
  • HuggingFace

Natural Language Processing:

  • spacy
  • langchain

Optical Character Recognition:

  • opencv-python
  • keras-ocr

Data Handling & Storage:

  • pandas
  • numpy
  • pinecone
  • requests
  • PyPDF2

Translation & Speech:

  • googletrans==3.1.0a0
  • gtts

Datasets:

  • Optical Character Recognition: Keras-OCR
  • Medicine: Kaggle

Diagram

The following diagram illustrates the workflow of IntelliCare:

IntelliCare Workflow

Explanation of Diagram

  • User Interaction: Users can interact with the system by uploading prescriptions and medical reports.
  • Data Extraction: Keras OCR is used to extract text from the uploaded images.
  • Model Training: Hugging Face models, fine-tuned with LoRA, generate descriptions based on the extracted text.
  • Embedding and Storage: Documents and prompts are embedded and stored in a vector database.
  • RAG System: The system augments and generates responses based on user queries by retrieving relevant information from the vector database.
  • User Responses: Users receive detailed and accurate information about their prescriptions and medical reports.

Built With

  • custom-searchapi
  • gemini-1.0-pro
  • generative-ai
  • google
  • keras
  • langchain
  • llm
  • lora
  • mixtral-8xb
  • ocr
  • pinecone
  • python
  • rag
  • streamlit
  • tensorflow
  • transformers
Share this project:

Updates