-
UI to swipe products
Inspiration
When shopping online, we often get stuck browsing through endless catalogs without finding clothes that truly match our style. We wanted to solve this problem by building a personalized clothing recommendation system that understands both the visual appeal of clothing (colors, textures, patterns) and the semantic meaning from product descriptions.
The inspiration came from noticing how humans decide on clothes using both vision and language: we look at the image and also read the description to check size, fit, or material. We wanted to replicate this behavior with deep learning.
What it does
Clozyt is a cloth recommendation system that suggests fashion items based on:
Image-based recommendations → Uses CNN embeddings (VGG16) to compare clothing images visually.
NLP-based recommendations → Uses Sentence-BERT embeddings to understand product descriptions and recommend semantically similar items.
In the future, these two signals can be combined into a hybrid recommender, giving users the best of both worlds.
How we built it
Image Feature-Based Recommendation
Preprocessed all product images.
Extracted embeddings using VGG16 (pretrained on ImageNet).
Computed cosine similarity between query image and catalog items.
Returned visually closest items.
NLP-Based Recommendation
Combined product metadata: name + category + description.
Encoded text using Sentence-BERT (all-MiniLM-L6-v2).
Measured similarity using normalized dot product.
Returned semantically closest items to a text query (e.g., “summer yoga tank top”).
Allows tuning balance between visual similarity and semantic meaning.
Challenges we ran into
Data quality issues → Some product descriptions were incomplete or noisy, making NLP embeddings less reliable.
Cold start problem → New items without images or descriptions couldn’t be recommended well.
Model efficiency → Extracting embeddings for a large catalog was computationally heavy.
Fusion tuning → Balancing visual and textual similarity was tricky without user feedback.
Accomplishments that we're proud of
Successfully built two independent recommendation systems (image-based + NLP-based).
Learned how to combine computer vision and NLP models for a single use case.
Designed a framework for a hybrid recommender that adapts to user needs.
What we learned
How CNNs (VGG16) can be repurposed for feature extraction in fashion similarity.
The importance of semantic embeddings from Sentence-BERT for text-based matching.
That recommendation quality improves drastically when combining multi-modal features (vision + language).
Handling vector similarity search efficiently with large datasets.
What's next for DevHacks-Clozyt-Visionistas
Hybrid recommender → Implement rank fusion of image + text similarity.
User personalization → Build GRU-based user encoders that learn preferences from browsing/purchase history. Real-time training: Everytime new products are added or new user are added the model gets trained automatically on new data.
Feedback loop → Adapt recommendations based on likes, skips, or purchases.
Log in or sign up for Devpost to join the conversation.