Diabetic Retinopathy Screening Tool

Diabetic Retinopathy affects millions, yet many patients suffer permanent damage due to delayed diagnosis. My project offers an accurate AI analysis of retinal images with explanation to everyone.

Comment 3

Inspiration

When looking at diseases affecting humans, Diabetic Retinopathy (DR) was quite shocking to me. It affects 10 million people in the US alone, and more than 100 million people in the world. Early detection of DR is very effective. Immediate treatment can stop the damage while it is minimal and prevent loss of vision. However, several factors limit access to DR screening. Detection of DR is difficult, sometimes even for trained and experienced doctors, which limits the number of specialists who can give exams. Additionally, there is a severe shortage of exam devices worldwide, and screening is further impeded by geographic barriers. Therefore, I set out to create an app that could assist medical professionals, speeding up diagnosis and leading to lower costs and more accessible exams.

What it does

The app's key features include: predictions about the DR stage, a heatmap, and an AI explanation. The DR model takes an image of the patient's retina and returns three key statistics. First is the severity value, which is how severe the model predicts the DR to be (0 is least severe, 4 is most severe). This is related to the class, which can be No DR, Mild, Moderate, Severe, or Proliferative DR. Probability indicates the confidence that the model is right, given by probability distribution of the SoftMax function. The heatmap is obtained from the class activation mapping (CAM) of the convolutional neural network (CNN), where gradients are calculated relative to a feature maps so the importance of each pixel is obtained. Red is the most important, blue is the least. Lastly, the app utilizes OpenAI calls to generate an AI explanation based on the image, heatmap, inference results, and optionally patient background information. The AI is meant to cover clinical assessment, heatmap analysis, key findings, patient explanation, recommendations, and monitoring. Retrieval augmented generation is also implemented, so users have the choice to add a selection of medical documents. At startup, a vector embedding storage is created and is maintained. The GPT model is able to reference medical literature in its analysis, lowering the chances of hallucinations.

How we built it

There are various Automated DR image assessment systems (ARIAS) online. I found several of these ARIAS that were public on GitHub. I wanted the model to assist medical professionals, but still believed that it was important to have a doctor check over the results. Therefore, I decided to go for an inherently interpretable model: generally CNNs, which can generate heatmaps to show what parts of an image the model is focusing on. This allows a professional to review the results and check that the model is detecting actual symptoms. I first found a CNN ARIA on GitHub and got it working. The repository I used was based off a Kaggle competition and had very high accuracy (97%). Afterwards, I added CAM capabilities, with PyTorch doing much of the work behind the scenes. I then created an AI analysis using GPT. I initially planned to use VQA, where a user could enter questions and get custom responses, but I did not have the time to do this. Instead, I wrote a prompt with patient data infused into it. I then created the RAG service. Using the OpenAI documentation, I used Streamlit to build the app, which was fast and efficient. AI Usage: ChatGPT was used for debugging. I also used GitHub Copilot autocomplete suggestions.

Challenges we ran into

Since I had only set up projects like this a few times before, getting the Conda environment working was a bit of a struggle. There were some pre-existing bugs in the code I obtained from GitHub, so I had to fix those. For example, there appeared to be a problem with using the GPU and CUDA for the device, so I had to use CPU. I also could not figure out how to use VS Code debug tools with Streamlit, and sometimes error statements were ambiguous or non-existent. This slowed me down but was not detrimental to my progress. The RAG and CAM parts were the trickiest. I understood all of the theory behind it, but actually getting the code working was challenging. Plenty of ChatGPT and Google were used here.

Accomplishments that we're proud of

I am most proud of the widespread applicability of the app. The GPT explanation could be understood by most patients, while the heatmap makes the app useful to medical professionals to speed up their diagnosis. I am also proud of the Grad-CAM, and how it was able to slot in quite well to the CNN architecture.

What we learned

I learned a great deal about using Streamlit. Sometimes the abstraction made the flow of logic hard to keep track of, but I learned to keep track of which process was running and how to use session states to manage the parts of the app. I learned more about CAM through seeing it work in real-time. I got better at debugging complicated programs and keeping track of what each file was doing.

What's next for Diabetic Retinopathy Screening Tool

The app shows the basics of what is possible for a DR detection app. In the future, I plan to refine the current utilities and possibly add new components, such as a chatbot I did not have time to implement. Eventually, I might write a paper on the app.

Built With

Share this project:

Updates