Deep Fashion Bot

Inspiration

Fashion can be seen as medium of self-expression. Unfortunately, some people don't have the resources to express themselves through this as much as others. Typically, this is due to the financial burden that comes from having to purchase stylish overpriced clothing. The Deep Fashion Bot aims to remove this barrier by providing suggestions for cheaper, yet boujee, attire.

What it does

The user sends a picture of clothing. The bot then uses a combination of web scraping and deep learning to find other articles of clothing in the internet that look similar to it. It messages the user back with links to these articles of clothing.

How I built it

Our pipeline consisted of three parts; web scraping, deep learning to get similar looking clothing and a messenger bot that relays information with the user. We used web scraping with scrapy to get information for various pieces of clothing several sites. The images of the clothing are downloaded and we use use a convolutional neural network (specifically with a ResNet architecture that Microsoft used to win ImageNet in 2015) to create a feature vector which was done on Microsoft Azure's cloud. When a user sends an image to our messenger bot (which is hosted on Heroku), we send the image url to the deep learning model which creates a new feature vector and compares it with all of the feature vectors in our database and returns the most similar pieces of clothing.

Challenges I ran into

Building web scraping scripts was a bit of a challenge. Because each clothing site organizes their content differently, we couldn't simply reuse the same scripting algorithm. We had to make small alterations to the algorithm each time. Another problem that we ran into was figuring out a way to deploy our work on Heroku while still being fast enough. To retrieve the clothing, we initially tried to use non deep-learning methods to create feature vectors (such as HOG, gabor filter, etc.). However, these methods although much quicker, could only find similarities in a limited scope such as shape similarity or color similarity. Using a convolutional neural network, we were able to overcome these challenges.