Curate

• 

  UI

• 

  For you page

• 

  Architecture

Inspiration

We wanted to make fashion discovery as effortless and engaging as swiping through a feed. Endless scrolling through e-commerce catalogs is overwhelming, so we set out to build a system that feels intuitive, fun, and truly personal.

What it does

Curate is a multi-modal, personalized fashion recommendation system that learns a user’s style in real-time. It creates a dynamic taste profile from both the visual appearance and textual descriptions of clothing items, ensuring recommendations are not only visually similar but also conceptually relevant.

Core Features:

Hybrid User Profile: Combines image embeddings (the look) and text embeddings (description, brand, category) into a single style vector.

Decaying Influence: Older likes/dislikes fade in importance, keeping recommendations current.

Diversity & Exploration: Uses MMR to avoid redundancy, cooldown penalties to prevent repeats, and occasional random exploration to surface new styles.

Dual-Dislike Handling: Short-term and long-term “anti-profiles” help the system learn not just what you love, but also what you don’t like.

How we built it

We scraped nearly 6,000 fashion products across multiple brands to build our dataset. For each item, we generated multi-modal embeddings using FashionCLIP, which combines the visual appearance of clothing with textual descriptions, brands, and categories.

On top of these embeddings, we implemented a personalized algorithm that updates a user’s taste profile in real time. Each swipe interaction adjusts a hybrid profile vector, a weighted blend of image and text embeddings , to better reflect evolving style preferences. We introduced decaying influence so that older likes fade while recent swipes carry more weight, along with anti-overfitting strategies like MMR-based diversity, cooldown penalties for repeats, and controlled exploration for novelty.

Finally, we connected everything into a polished interface: a React-based swipe experience powered by a FastAPI backend, ensuring users receive dynamic, personalized recommendations with every interaction.

Challenges we ran into

One of the biggest hurdles was experimenting with the Two-Tower recommendation architecture for retrieval and ranking. While we reviewed open source solutions hoping to adapt them to handle multi-modal embeddings (images + text) and real-time swipe data, it proved more complex than the hackathon timeframe allowed.

Accomplishments that we're proud of

We built a fully functional swipe-based fashion recommender from scratch. On the data side, we scraped and processed ~6,000 clothing items and generated multi-modal embeddings with FashionCLIP, capturing both visuals and text.

We then designed a personalized algorithm that updates a user’s taste profile in real time with features like decaying influence, MMR-based diversity, cooldowns, and controlled exploration to keep recommendations dynamic.Finally, we delivered a polished React frontend with smooth swipe animations, backed by a FastAPI backend serving recommendations seamlessly end-to-end.

What we learned

We learned how to apply multi-modal embeddings from FashionCLIP to capture both visual and textual aspects of clothing. We gained experience in building real-time personalization algorithms, and saw firsthand how diversity and exploration techniques can keep recommendations engaging. Most importantly, we learned the trade-offs of designing recommendation systems that feel both accurate and surprising.

What's next for Curate

Our next steps focus on scaling Curate into a production-ready system. We’ll train the model on an even larger dataset to strengthen personalization, add onboarding flows to solve cold-start issues, and expand the dataset across more categories, brands, and seasons. On the modeling side, we plan to move beyond our current personalized algorithm by exploring Two-Tower networks for scalable retrieval and VLMs based user models to capture evolving style sequences. To keep results both relevant and fresh, we’ll experiment with vector databases to deliver low-latency and faster retrieval.

Built With

• fashion-clip
• numpy
• pandas
• pillow
• pytorch
• react
• scikit-learn
• tqdm
• transformers
• typescript