Fit Finder

The Outfit Search Engine

Comment 5

Inspiration

Fit Finder is born from a deep concern about the unsustainable nature of fast fashion. Millions of tons of textile waste end up in landfills each year, often from impulse purchases worn only a handful of times.

We recognize that a significant contributor to this problem is the "friction of understanding" one's own wardrobe. People often buy new clothes because they can't easily visualize how to mimic new outfits with their existing items.

Fit Finder aims to eliminate this friction by providing a seamless and intuitive way for users to explore their closet with a fresh perspective.

What it does

Fit Finder acts as a powerful search engine, helping you recreate popular outfits you see daily on platforms like Pinterest using your own clothes! It also provides easy, live-updated sustainable buying options if you still wish to purchase new items.

  • Wardrobe Digitization: Users can easily add their clothing items to their digital wardrobe (or "closet") within Fit Finder.

  • Outfit Search: When you come across an outfit you like – perhaps on social media or Pinterest – you can simply paste an image into Fit Finder and search.

  • Intelligent Vector Search: Our Flask backend, powered by OPEN CLIP, a vector embedding generation model, converts your clothes into vectors for semantic search. These embeddings are stored in MongoDB, allowing for semantic searching in addition to regular color based searching (hybrid search).

  • Outfit Recreation: Fit Finder then performs a complex similarity search across your wardrobe's vector embeddings, identifying the closest matching items to the components of the target outfit. Our intelligent algorithm understands nuances in style and garment type beyond simple pixel matching to determine if you can realistically recreate the look.

  • Sustainable Shopping: Fit Finder helps in recreating the outfit using your existing clothes, alongside easy, live-updated sustainable shopping results for similar products from known ethical brands and information on sustainable cloth materials if you choose to purchase.

How we built it

Our app is structured around a three-part architecture:

  • Flask Backend Server: This is where the core logic resides. We chose Flask to build the API.
    • Searching: Upon receiving a query image, the backend analyzes it, extracts the dominant color for initial filtering, and generates a vector embedding. It then performs a hybrid search (vector search AND filtering) in the MongoDB database, leveraging both color and vector similarity for more accurate results.
    • Embedding Generation: We are using OPEN CLIP provided by ChromaDB for generating vector embeddings of clothing items, enabling accurate semantic search.
    • Vector Database (MongoDB): We utilize MongoDB's powerful vector search capabilities to store and efficiently query the embeddings of the user's wardrobe. Each clothing item is represented as a document containing metadata (e.g., description, color) and its corresponding vector embedding.
    • Google Gemini Integration (Enhancements): We integrate Google Gemini to identify individual clothing items within the query image, leading to more precise search results! Something which may require dedicated segmentation models like YOLO can be done on the fly by Gemini!
  • Perplexity AI for shopping results: Perplexity excels at finding relevant search results. We run a Perplexity search using a description of the query image generated by Gemini to find similar products. We then analyze the search results to provide a sustainability score based on the brand's known efforts and the fabric composition (e.g., prioritizing 100% cotton).
  • Web Extension (Half-Baked - Proof of Concept): Allows the user to select any image on a website and automatically open a search on our website for ease-of-access. This would streamline the process the user would take instead of having to save and upload the image themselves.

Challenges we ran into

  • University Wi-Fi restrictions: A significant roadblock was the university's Wi-Fi network, which unexpectedly blocked outgoing requests to our MongoDB Atlas cluster. We resolved this by switching to eduroam.
  • Open CLIP limitations: Running searches relying solely on vector matching with OPEN CLIP yielded slightly inaccurate results because the model isn't specifically trained on detailed fashion data.
  • Perplexity Sonar has no vision capability: Perplexity has no support for images in prompts thus we had Gemini generate a detailed description of the cloth article to run the perplexity search, which has great results!

Accomplishments that we're proud of

We're incredibly proud of several aspects of Fit Finder:

  • Using Google Gemini to extract clothes from an outfit: Getting detailed crops of the query image and then using them to search makes our search much more powerful, significantly improving the results.

  • How to make vector databases: This is something we all wanted to work on as vector DB' s are used in RAGs, we found an interesting application of them in this case and we are proud of doing this so fast!

  • Addressing a Personal Problem: This is a personal issue that we face ourselves. We're proud that we can create a project so relevant to us as we would like to break the notion of search Temu or TJMaxx for outfits that we may already have.

What we learned

Through the development of Fit Finder, we gained valuable insights into:

  • Mongo DB vector search: MongoDB vector search was very quick and easy to setup to our surprise!
  • The perplexity API: We got free credits which helped us learn it.
  • Chrome extension API: We learnt how to create an use the basics of chrome extensions and the basic structure they all have.

What's next for Fit Finder

  • Enhanced Browser Extension: Develop a fully functional browser extension enabling one-click outfit image selection and quick searches from any webpage, along with features for saving favorite outfits and tracking wardrobe items.
  • Integration with eBay (Second-hand Clothes): Explore using the eBay API to enable image-based searches for second-hand clothing options.
  • Improved Embedding Model Integration: Continuously evaluate and integrate more accurate and robust embedding models beyond OPEN CLIP to further enhance the accuracy of outfit matching.
  • Complete Chrome Extension: Complete the Chrome extension and possibly a Firefox extension that would allow users to seamlessly interact with our project.
Share this project:

Updates