Inspiration
When Inditex proposed us this challenge about finding and filtering similar images from a dataset, we were really motivated to see what we would be able to create. We have been interested in how neural networks work and how to use an open source model to approach this task, furthermore we are really interested in image visualization and how graphics work in a computer so we thought it was a great oportunity to learn about this topics in an intensive and short journey.
What it does
Pick&Fit is a tool that helps you pick similar clothes and other items in a fun way. You can choose a random item from an Inditex brand and the AI will show you similar items related to that. So it's a fun and random way to see different clothes and get to navigate through the pieces in a different way you are used to.
How we built it
We imported an opensource VIT model from huggingface and we generated embedding in a batch of images from the Inditex dataset that we had previously processed. Due to the presence of duplicate images and URLs in the dataset, we developed an algorithm to eliminate them as well as the invalid ones. With the VIT model (google/vit-base-patch16-224) and the embeddings created, we keep them in a vector structure using the FAISS library, that way we have all the embeddings processed and organized by similarity and proximity. So when a random query is given to our program it calculates it's embedding and it searches for the similar embeddings (the vectors with the lowest distance from the query), and returns the five closest ones. To display the pieces and make it functional we also developed a website with React that fetches the python program to display the data and also let's you generate new random pieces.
Challenges we ran into
Taking on the task without prior experience in Python, CNNs, or deep learning posed a significant challenge for us. We had to start from scratch and encountered numerous problems. Despite getting stuck in various places, due to our limited knowledge, we remained committed to learning. Our biggest challenge was to interpretate the embeddings generated with our ViT model, and getting the hang of how tokens and outputs work. We also had problems with working with data structures with python, especially with the datasets and the faiss structure.
Accomplishments that we're proud of
We are really proud of all we've learned and accomplished over the weekend. Despite being through lots of challenges, everything we've implemented this weekend we had never done it before. Taking advantage of that opportunity makes us very proud of ourselves.
What we learned
We have a global idea of how deep learning works and the process behind the pipeline of ViT. We created our first programs with Python. We learned how to start programming a front end and learning how to connect our backend with our front end.
What's next for Pick&Fit
We're really happy with the results but we have a bittersweet mouthtaste because we wanted to fine-tune our model with our data to improve its performance. We believe this would significantly improve the similarity outcomes.
Log in or sign up for Devpost to join the conversation.