SkinSight

Inspiration

As a group of teenage girls we are always looking for an outlet to assure us what our skin conditions are without the fear of googling and being told that it is cancer. The inspiration to create an effective solution for this problem comes from our collective desire for accurate skin analysis. As a team of women of color, we know the struggles of finding the correct foundation or concealer shades at the makeup shop due to lack of inclusivity in beauty brands. Different skin problems look different on various skin tones. This is also important for medical professionals when identifying skin problems.

What it does

Our application detects common skin problems on diverse human skin. For now, the 11 skin problems our deep learning model was trained to detect are blackheads, cold sores, cysts, eczema, hives, moles, psoriasis, rosacea, scars, vitiligo, and whiteheads.

How we built it

We used deep learning and image classification on a series of images we compiled from the internet. We gathered 5 images for each of our desired skin problems in 3 skin shades, Fair, Medium, and Dark. In total, we had 162 images for our deep learning model to train with. After that we utilized YoloV5 and OpenCV in PyCharm to construct our computer vision and facial recognition. OpenCV is what we are using to create a live camera feed and process captured images. YoloV5 is being used for object detection for facial recognition and condition detection. Our model utilized a convolutional neural network to train the model, which was done with the help of the sequential and dense layers from the keras package. In order for the neural net to read the labels, we had to hot-encode our labels for each skin problem using scikit-Learn.

Challenges we ran into

We attempted to use Tensorflow for the purposes of our model training initially, and we were unable to connect our model and our computer vision appropriately. We searched for alternatives and determined that OpenCV was the best.

The main roadblock we ran into was getting our camera to work with OpenCV, we spent around 3 hours debugging this issue. We had to figure out which index to use for the camera as our final goal was to be able to use a cell phone camera to scan a face and the program be able to tell the user what skin problem they have.

Another issue we faced was finding photos for different skin colors. Most images on Google were skin problems of fair skin tones. So, this also took us quite a bit of time to put together. This leads to the issue of ethics. We initially wanted to find images on open sources however, the search proved to be difficult on Google itself and time was running out.

Accomplishments that we're proud of

According to the model.evaluate() method from keras, we had a 62% accuracy. We are only freshmen currently, and we have never done computer vision or image classification before, so the fact that we were able to pull it together in just 24 hours is kind of a slay.

What we learned

How to open a live feed in PyCharm, how to recognize faces using computer vision, how to train a model for image classification, how to test and train the model, how to link the model to the computer vision.

What's next for SkInsight

The next steps for SkInSight would be to improve our model by training it further. We want to gather even more images for an even bigger number of skin shades than the ones we were able to gather within these 24 hours. We also want to be able to account for different ranges of lighting. Another step we want to take is improving our ethics by taking more time to find images that are from open sources or even better by taking our own photos with the consent of the people being photographed.