I constantly think about how computer vision could help people. My original plan was to train a convolutional neural network to detect skin diseases and connect it to mobile app. However, that's been done by teams with much more resources, expertise, and experience. Furthermore, the cost of misclassifying instances of skin cancer is very high.

Later I thought that doing the same with mold on household surfaces would help people who lack the financial resources to hire a mold inspector. Also, the cost of misclassification would be low as it would not harm the user.

What it does

MoldAI classifies images of mold on household surfaces and displays information about it.

How I built it

I used google's image search api to collect images of various species of mold on walls and other surfaces. I manually went though the data to discard irrelevant results and selected the most common strains.

For the machine learning part I trained a pytorch model to detect the 7 most common strains using transfer learning. To do this I used features extracted by a pytorch implementation of EfficientNet (

Once I had the model I put it in a python flask server which takes image data submitted through an html form and feeds it into a classifier. The classifier then outputs a list of predictions which are sent to the user along with some information about the detected strains.

Challenges I ran into

Getting the necessary data to train a machine learning model was a major challenge. Finding a neural network architecture that could reliably output predictions with so little data was also a challenge. Creating a frontend was especially difficult given my lack of experience and time constraints.

Accomplishments that I'm proud of

I am very proud that I managed to get 71% accuracy on my validation set considering how small my data set was (311 images). I am also pretty stoked that I managed to connect my machine learning model to a usable frontend.

What I learned

I learned how to do transfer learning with EfficientNet and how to put a machine learning model into a usable application.

What's next for MoldAI

I will collect more data to train a better model and create a frontend in React.

Share this project: