LLaVA-Med-Retina

• 

  LLaVA-Med-Retina Case 1 (Correct)

• 

  LLaVA-Med Case 1 (Incorrect)

• 

  LLaVA-Med-Retina Case 2 (Partially correct)

• 

  LLaVA-Med Case 2 (Incorrect)

• 

  LLaVA-Med-Retina Case 3 (Partially correct)

• 

  LLaVA-Med Case 3 (Incorrect)

• 

  LLaVA-Med-Retina Different Domain 1 (Expected)

• 

  LLaVA-Med-Retina Different Domain 2 (Expected)

• 

  DDR Q&A Instruct Generation

• 

  LLaVA-Med-Retina Training

Inspiration

Fundoscopy: A Window to Health

Fundoscopy, a simple eye exam using an ophthalmoscope, offers a powerful glimpse into many aspects of patient overall health. By examining the retina, the doctor can assess not just eye diseases, but also detect signs of diabetes, hypertension, and even neurological issues. This painless and lightning-fast procedure is particularly valuable in resource-limited settings. Mobile healthcare crews equipped with portable fundus cameras can quickly screen large populations in developing countries and remote communities. Early detection of these conditions allows for timely intervention, significantly improving healthcare outcomes and reducing the burden on already strained medical systems - directly addressing UN SDG Goal 3, "Good health and well-being".

Generative Models: Adapting to Unseen Data

When it comes to analyzing medical images, traditional discriminative models are good at tasks they've been trained for. However, they struggle with unseen data, thus presenting a major limitation in real world applicability. Here's where generative multimodal models shine, with their ability to seamlessly handle downstream tasks. This could involve doctors simply describing specific retinal characteristics of a new population (e.g., rarely contacted tribes) and the model can adapt accordingly. This eliminates the need for extensive data collection and retraining.

Enriching Data with Conversations

Imagine a vast database of retinal images paired with medical descriptions. Large language models (LLMs) can unlock hidden knowledge from this data. LLMs can analyze the captions and generate conversations that elaborate on the medical descriptions. These conversations can explore different scenarios, discuss potential complications, and highlight relevant medical history. This rich, conversational text data can then be used to train a combined vision and language model, leading to a more comprehensive understanding of the link between retinal features and health conditions.

What it does

LLaVA-Med-Retina is a large language and vision model specializing in retinal images. At current stage it is able to recognize fundus images from various domains it was pre-trained on and conduct basic reasoning on the topic of diabetic retinopathy (DR) signs presence. It is able to recognize, although not on a consistent basis, the presence of various lesions associated with DR such as hemorrhages and exudates. It also shows promising sings of disease grading capabilities.

How we built it

Data collection

We adapted the DDR dataset ("https://doi.org/10.1016/j.ins.2019.06.011") containing images of healthy and sick (DR) patients and associated annotations (0-5 severity grade, lesion bounding boxes and segmentation masks) to introduce manually obtained text descriptions (compressing label data into text format) and subsequently generate rich Q&A / Instruct data by utilizing OpenAI GPT4 (simulate conversations with the user inquiring about the image details and the supposed model answering, while operating solely on the text description). The final dataset contains 1K image - dialog pairs.

LLaVA-Med fine-tuning on DDR Q&A dataset

LLaVA-Med was created by Microsoft based on LLaVA (Vicuna LLM + BioMedCLIP vision encoder) by pre-training on wide array of images and captions taken from PubMed. We fine tune this model on our DDR Q&A dataset by deploying 8xA40 rig on RunPod.io. The trained model is served using the Gradio app similar to LLaVA-Med.

Challenges we ran into

• $35:)

Accomplishments that we're proud of

• !Fine-tuned working 7B multimodal model in less than 24 hours!
• Novel DDR Q&A / instruct dataset

What we learned

• First time training large multimodal models

What's next for LLaVA-Med Retina

• Large scale fine-tuning on Retina Image Bank (~30,000 image - captions covering diverse conditions) with LLM-augmented instruct data

Built With

• biomedclip
• gemini
• gpt4
• gradio
• huggingface
• llama
• llava
• llava-med
• pytorch
• runpod.io
• transformers
• vicuna