FitCheck

FitCheck brings your closet into the fitting room. Scan a garment to discover new outfits, spot duplicates, and get a purchase verdict—so you buy less and wear more. Use FitCheck. Shop Sustainably.

Comment

Inspiration

We've all come home excited about a new piece of clothing, only to realize it goes with almost nothing we own. It ends up barely worn, which wastes money and adds to textile waste. How often a garment gets worn is one of the biggest factors in its footprint, and a 2017 Ellen MacArthur Foundation report found that clothes in the US are worn about a quarter as often as the global average.

We built FitCheck to answer one question before checkout: will I actually wear this with what I already own? By bringing your closet into the store with you, we want to help people buy clothes they'll use and build a style that feels personal.

What it does

Think of FitCheck as a stylist who has seen your whole closet. Most fashion apps ask whether something looks good. FitCheck asks whether it's worth owning.

  • Build your closet from one photo. Lay clothes on your bed and snap a picture. FitCheck finds, cuts out, and tags each garment, and turns it into a clean catalog image. You review and correct the tags before anything is saved.
  • "Should I buy it?" Photograph an item in the store. In 5 to 20 seconds you get BUY, CONSIDER, or SKIP with a 0 to 100 score, the top two reasons, the new outfits it unlocks, its cost per wear, and a warning if you already own something just like it.
  • Sustainability grade (A to E). CO2 and water per wear, based on the fabric and how often you'd actually wear it.
  • Better options, not just "no." A SKIP shows real alternatives that score higher, and a BUY shows pieces that pair with it. All of them go through the same scoring.

How we built it

We began by brainstorming in Notability and bouncing ideas off chatbots to create a shortlist of potential projects. We outlined each idea's pros and cons, independently considered the most promising directions, and asked a chatbot to act as a devil's advocate.

  1. Detect and tag (Gemini 3 Flash): one call per photo returns every garment's bounding box, attributes, and estimated price.
  2. Cut out (SegFormer): masks each garment, with a quality gate that falls back to a plain crop.
  3. Catalog image (Gemini 3.1 Flash Image): redraws each cutout as a product shot and checks it three ways.
  4. Embed (FashionCLIP + FAISS): catches "you already own this" using image and text similarity.
  5. Judge outfits (OutfitTransformer, 0.922 AUC in our evaluation): a slot-by-slot outfit search, calibrated for each outfit size, so "unlocks 7 outfits" is a real count.
  6. Decide (pure math): 40% versatility, 20% outfit quality, 40% cost per wear against your closet's median. Near-duplicates are capped at SKIP.
  7. Shop (Gemini + Google Search grounding, with live search of about 16 storefronts as a fallback): every product is scored by the same pipeline.
  8. Sustainability (Python): computes CO2e and water per wear from sourced fiber factors and predicted wears.

Stack: Next.js, React, TypeScript, and Tailwind on the frontend; FastAPI, PyTorch, Hugging Face Transformers, FAISS, SQLite, and google-genai on the backend; tested with pytest and deployed on Vercel.

Challenges we ran into

  • Finding models suited to our use case was one of our biggest challenges. Recognizing a garment and deciding whether garments look good together are different problems. FashionCLIP helped with similarity, but outfit compatibility required a separate approach.
  • OutfitTransformer's raw scores behave differently depending on the number of garments in an outfit, so we calibrated scores across outfit sizes. We also refined the final verdict's balance of versatility, outfit quality, cost per wear, and redundancy. This involved tuning application logic and thresholds rather than retraining model weights.
  • Sustainability introduced uncertainty of its own. A photo cannot reliably establish a garment's exact material composition or how often someone will wear it. We therefore present environmental impact as an estimate, explain its assumptions, and keep it separate from the purchase verdict.
  • Deployment also required significant work because the backend runs several large machine-learning models. We used CPU-focused dependencies, fetched model weights during the build, and adapted the application to Vercel's filesystem constraints. Temporary runtime storage made the hackathon deployment possible, while durable persistence remains a next step.

Accomplishments that we're proud of

  • We're proud of building a complete pipeline from a raw photo to an editable, visually organized digital wardrobe. FitCheck identifies garments, isolates them, and captures useful attributes while giving users the ability to correct the results.
  • We're especially pleased with the outfit combinations it produces. Combining category-based rules with compatibility scoring helped us create looks that felt cohesive in our testing. Extending that process to retailer products also allowed us to evaluate potential additions against the clothes a user already owns.
  • Another accomplishment is the nuance behind Buy, Consider, or Skip. The recommendation considers what the garment contributes to a particular wardrobe: the outfits it unlocks, its similarity to existing pieces, its quality of matches, and its estimated value over time. We turned those signals into explanations users can understand.
  • Finally, we're proud of making sustainability part of the decision. FitCheck helps users see how materials, expected wear, and redundancy influence its estimates, encouraging them to think about the life of a garment beyond the purchase.

What we learned

  • We learned how to coordinate a multimodal AI pipeline, connecting detection, segmentation, structured attributes, embeddings, and compatibility scoring. Integrating several models taught us how strongly each stage depends on the quality of the previous one, and why fallbacks are essential when a model or API fails.
  • Working with FashionCLIP deepened our understanding of image and text embeddings, cosine similarity, and vector search through FAISS. We also learned to distinguish visual similarity from outfit compatibility: similar pieces may duplicate what someone owns, while visually different garments can complement each other.
  • Using open-source models taught us how much adaptation happens outside the model itself. Preprocessing, score calibration, threshold tuning, and application logic were central to making these tools useful for FitCheck. We learned to evaluate model outputs in the context of the actual user decision.
  • Deploying through Vercel also showed us how hosting an ML-powered application differs from hosting a typical website. We gained experience managing large dependencies, model weights, startup costs, and storage constraints, and a clearer understanding of what the application needs beyond a hackathon demo.

What's next for FitCheck

  • Help getting dressed every day: pick one piece and get outfits built around the weather, season, and occasion. A white T-shirt might go with shorts on a summer afternoon, or under a puffer jacket in winter.
  • Learning your taste: "I'd wear this" and "Not my style" feedback that sharpens future verdicts and suggestions through reinforcement learning.
  • Real cost per wear: automatic closet updates from receipts, and tracking what you actually wear.
  • Closing the loop: resale and donation suggestions for clothes that never get worn.

Our goal is for FitCheck to help you decide what to buy and what to wear.

Built With

Share this project:

Updates

Submission history