SkinScan

Inspiration

Melanoma is the most serious type of skin cancer, and statistics show it is increasing. The five-year survival rate is high when detected early (98.4% for localized melanoma), but decreases significantly with later stages. In addition, early detection is often delayed due to inaccessibility, high costs, or simply not knowing when a mole or mark is worth checking. This promoted the idea of SkinScan, a way for you to check that bump easily without paying a single dime.

What it does

SkinScan allows users to upload a photo of a skin lesion, mole, or mark, and instantly get a prediction of whether it’s more likely to be benign or malignant based on visual similarity to a curated dataset of labeled examples. Users can get information instantly due to the model already running thus allowing for fast lookups. However, the backend has not been deployed so it won't work on anyone else's computer.

How I built it

I first conducted exploratory data analysis using Python and its various libraries like Pandas, Matplotlib, Seaborn, and Numpy. Next I built the application using a React frontend with TypeScript and TailwindCSS with a FastAPI backend that has the main logic of finding out whether an image of skin is benign or malignant. The Python script uses OpenAI's CLIP model which I trained with the image data.

Challenges I ran into

One of the biggest challenges that I ran into was trying to use computer vision in order to analyze the different skin lesions in the 400,000+ images. The problem was that the dataset was heavily imbalanced and thus using computer vision caused the results to be skewed when doing a train test split as there were more than 400,000 benign images compared to the 300 malignant images as well as the 100 indeterminant images. Thus computer vision would never work due to images not being consistent and I had to pivot in a different direction; as a result, I found out about OpenAI's CLIP model which learns to associate images and text by training on a massive dataset of image-text pairs, enabling it to understand the relationship between visuals and their descriptions.

Accomplishments that we're proud of

One of my biggest accomplishments was getting OpenAI's CLIP model and architecture to work with my personal model of images and the feature data that I was using since the data that was given was heavily imbalanced. In addition, combining the frontend and the backend was extremely difficult as you are not allowed to run a Python script within the browser (for obvious security reasons). So trying to circumvent that issue was particularly difficult but in the end I just had to create a web server using FastAPI that connected my Python script to my React frontend. In addition, trying to solo a project is not the easiest yet you gain so much knowledge and experience working with new tools.

What's next for SkinScan

My original goal was to also have a separate tab dedicated to finding local hospitals using the Melissa API or Google Maps API. Unfortunately I ran out of time before implementing it but I believe that this would be the right step in raising awareness about the dangers melanoma can have and the possibilities of saving lives by treating it early instead of risking injury later. In addition, I didn't have the time to add a feature that would allow users to take pictures using their phone or their camera directly. Also, the backend only runs locally since I couldn't figure out how to deploy it nicely. These are some of the features that I would have liked to implement if I had more time but oh well :(