ClostAI

Inspiration

A French friend told me Americans don't dress very well. I took it personally, but I honestly don't know much about fashion. I wanted a way to see how clothes look on me before I spend money on them, and to borrow a look I like without guessing. ClosetAI is my attempt at that.

What it does

• Closet: Add clothes and see them on yourself through the camera in real time.
• Live try-on: A pose-tracking overlay fits a garment to your shoulders and torso as you move. It's labeled as a preview, not a final render.
• High-quality render: A snapshot is sent to IDM-VTON on Replicate for a more realistic result.
• Voice assistant: Built on Moss and a LiveKit Python agent. It can search your closet, suggest outfits, start a try-on, and remember preferences.
• Dress like / Shop the look: Name a celebrity or style. The app builds a sourced style profile, matches it to your closet, and lists the pieces you're missing.
• Try before you buy: Products can be tried on before they're saved. Checkout requires a separate confirmation step.

How I built it

• Frontend: Next.js and React, with a Web Worker running MediaPipe Pose Landmarker for tracking.
• Try-on backend: Route handlers validate uploads, preprocess images, and start Replicate predictions. Jobs are stored in Upstash Redis and polled from the browser. Finished images go to Supabase Storage and are cached by input hash.
• Style and recommendations: Claude is called only from the server, with schema validation that rejects outfit items that aren't in the closet. Style profiles use cited web sources, and a profile is only accepted with at least two distinct sources.
• Voice: A Python worker deployed through Moss, with seven function tools that call authenticated routes in the Next.js app.
• Demo mode: Cached audio, voice responses, pose landmarks, and try-on results let the full demo run offline.

Challenges

• The live overlay looked pasted on. A 2D warp has no occlusion, so garments drew over arms and hair. Fixing that properly means segmentation and a mesh warp, which is the next thing I'm working on.
• Garment cutouts. Product and closet photos have backgrounds, hangers, and hands. Clean cutouts are needed for both the live view and the render.
• Async jobs. Replicate takes 10 to 30 seconds, so the app returns a job ID right away and polls for the result. Duplicate requests for the same images share one job, and cache results are never overwritten by a stale status update.
• Race conditions. Voice turns, try-on jobs, and preference writes could conflict when they overlap. I listed them in RACE_CONDITIONS.md and added guards for each.
• Cost control. Rate limits and cached results keep paid API calls predictable during testing.
• Secrets. API keys stay server-side, and the browser never sees them.

What I learned

• A good demo depends on the failure path as much as the happy path. Every feature needs a clear fallback.
• Real-time computer vision is mostly about stability: smoothing, confidence thresholds, and holding the last good pose.
• Keeping an LLM inside a validated schema, and rejecting any output that refers to data that doesn't exist, makes the results much more trustworthy.
• Checkout has to be built around explicit user consent. Showing the cart, total, and payment method before any charge matters more than the payment code itself.

What's next

• Finish occlusion-aware live rendering so the overlay sits behind the body.
• Persist closet photos so items survive a refresh.
• Connect a real Visa checkout adapter, or keep the test-payment label visible in the demo.

Built With

• moss
• solo
• visa