CloseTheLook

• 

  webpage

• 

  shoping

• 

  outfit-detected

Inspiration

Fashion content is one of the biggest drivers of impulse shopping, but the way people actually act on it is broken. You see an idol's outfit in a livestream, screenshot it, and then burn twenty minutes reverse-image-searching pieces across five different shopping sites — only to find out later that you already owned something close enough in your own closet the whole time. We wanted to close that gap between "I want this look" and "I own most of this look already" in a single tap, instead of a browser full of tabs.

What it does

You're watching your favorite idol perform live, and you love their outfit. Instead of scrolling through five different shopping sites trying to piece it together, our app watches with you. In real time, it identifies every piece they're wearing, checks it against your own closet, and tells you: "You already own 90% of this look — you just need one jacket." Then it shows you where to get that exact piece for under $40. We're not another AI shopping assistant. We're the first app that tells you to shop your own closet before you shop anyone else's — and only sends you to buy the one piece that's actually missing.

How we built it

We split the build into a React (Vite + Tailwind) frontend and a Node.js/Express backend, and worked the pipeline stage by stage:

• Live video — a Vonage Video API session powers the broadcaster/viewer experience, with a pre-recorded video fallback so the demo isn't hostage to live-stream reliability during judging.
• Frame capture — on "Recreate This Look," we draw the current <video> frame onto a hidden canvas and ship it as a base64 JPEG.
• Outfit detection — the frame goes to Google's Gemini vision API with a structured-output prompt (responseMimeType: application/json) that returns each garment's category, color, material, pattern, and silhouette — no markdown, no guesswork parsing.
• Matching engine — rather than reaching for embeddings on day one, we built a weighted rule-based scorer (category 30%, color 30%, material 20%, silhouette 10%, pattern 10%), with color-family grouping (e.g. "charcoal" and "black" count as a match) so near-misses still score fairly.
• Shop the gap — anything scoring below threshold is flagged missing, and we surface 3–5 shoppable alternatives from a seeded product dataset, sorted by similarity.
• UI pipeline — a dedicated screen per stage (Live Stream → Analyzing → Outfit Detected → Closet Match → Missing Item → Shop Modal), driven by a single usePipeline hook so state flows cleanly from frame capture through to the shopping cards.

We deliberately built the matching engine and mock data first — before touching Gemini or Vonage — so we always had a working, demoable app end-to-end, and layered the real APIs on top.

Challenges we ran into

Integrating the APIs was the hard part. Gemini's vision responses needed careful prompting to stay reliably valid JSON rather than drifting into prose or markdown fences, so we forced responseMimeType: application/json and built our schema defensively around occasional missing fields. Vonage's Video API required real HTTPS to get camera/mic permissions working at all — Chrome silently refuses getUserMedia on any non-localhost, non-HTTPS origin — so we had to wire up a self-signed dev certificate just to test the broadcaster/viewer flow across two devices on the same network. We also had to reconcile mismatches between Gemini's own category vocabulary (e.g. it can return both "jacket" and "outerwear" for the same garment type) and our closet's categories, which meant adding alias handling into the matcher rather than trusting a strict equality check.

Accomplishments that we're proud of

We're proud that the full pipeline — live capture, AI detection, closet matching, and shopping recommendations — works end-to-end on real API responses, not just canned demo data. The matching engine holds up well even on imperfect Gemini output because of the color-grouping and category-alias logic, and the fallback video path means the demo stays reliable even if live video misbehaves in front of judges. We also like that the core idea reframes AI shopping around what you already own instead of just pushing more purchases.

What we learned

We learned how much of an "AI feature" actually lives in the boring plumbing around the model — structured-output prompting, defensive JSON parsing, and picking a fast rule-based baseline before reaching for embeddings. We also learned a lot about the browser security constraints around live video (HTTPS, permissions, LAN testing) that don't show up until you try to demo across two real devices, and how valuable it is to build the UI against mock data first so API integration never blocks forward progress.

What's next for CloseTheLook

• Upgrade the matching engine from rule-based scoring to Gemini text embeddings + cosine similarity for more nuanced, "AI-native" matches.
• A real closet upload flow (photo → auto-tagged attributes) instead of a seeded dataset.
• A side-by-side recap view: Idol Look → Your Closet → Your Final Look composite.
• Real retailer integration for shopping alternatives, replacing the mocked product catalog.
• Persisting user closets and match history so the app improves the more you use it.

Built With

• gemini
• raect.js
• vonage