Fashion Recommender

The inference pipeline(s) of the model
The pre-processing of the model

Inspiration

We are a team of data scientists inspired by the latest advances in the field of machine learning and AI. Scalability and accuracy are key and we surpass the best models over and over again. The opportunity to work on a project for the global retailer Inditex, which is one of the most successful and inspiring companies in our country, and apply our knowledge, was the best we would ever foresaw when coming here.

What it does

We built a full-stack application for visualizing, filtering and recommending the products of Inditex's dataset. In our webpage, you can find the lastest fashion clothing from its features (type, color, category, pattern, ...). Furthermore, given a product of your liking, we will recommend you the most similar alternatives you might consider to find exactly what you are looking for.

How we built it

The back-end is a model that was built in two phases.

Pre-processing: First, we used an image segmentation model for identifying the regions of interest of the image, containing the product. Then, we get rid of the other part of the image and replace it with a green screen. Given the masked image, we feed it to a CLIP model to gather its embedding (the mask helps the model focus on the product). Finally we store all of the (normalized) embeddings in MongoDB for front-end inference and finding the most similar images in less than a second.
Inference: with the embeddings of the images and the MongoDB algorithm to find the closest images, we can now use the CLIP model to get the embedding of text and find the closest images. Furthermore, given an image, we can find the k that are closest to it.

The pre-process is key for inference efficiency and scalability and all models are fine-tuned for fashion.

The front-end is tailor-made with Next and connected to the MongoDB database. It gives the user the opportunity to easily navigate and visualize the amazing performance of our model.

Challenges we ran into

Firstly, as a team of data scientists, front-end was a big challenge coming into the hackathon. With little to no experience in building a website, we wanted to deliver the best possible front-end. To do so, we didn't want to limitate ourselves to simple frameworks and exploited the experience of one of our members in using the Next framework in a previous project.

Processing the dataset was a challenge itself as we were blocked by DDoS protection when trying to downolad 140k images 1 by 1. Saving it locally without taking up too much storage also took some tries. On the other hand, having such a big dataset meant that training and inference were slow in a challenge where time is gold.

Finally, accuracy and scalability are key for the project. That's why we tested 5 different models for the embeddings, only keeping the one with the highest accuracy (we leave in our GitHub repository the noetbooks of the models that we tested).

Accomplishments that we're proud of

In the end, we are extremely proud of our front-end, as it is integrated seamlessly with the back-end and offers the user an intuitive and appealing interface.

The model, shows incredible accuracy, exceeding any expectations. And we even managed to merge it with some text embeddings.

Not only that, we participate too in the challenges of MongoDB and Replexica, as we have been able to use them to notably improve the quality of our project.

What's more, we are proud of our code, which achieved everything we intended while being well-structured, documented and easy to read.

What we learned

We definetely learned a lot about front-end, full-stack integration, state-of-the-art models, AI pipelines, MongoDB, Replexica and some graphic design to make it look good!

What's next for Fashion Recommender

We would love to be able to do connect the front-end directly to the model through an API, thus allowing to process any text query.

Moreover, with more time we think we could have maybe fine-tuned the models better by following the definition of loss we used in the training of the Siamese CNN (which we didn't use in the end).

We are also sure that an experienced front-end developer has a lot to suggest to level up our website.

Finally, the project could be deployed in an always-functional domain (right now you'll have to run the website locally or try it out with us).