Aura TryOn

Inspiration

Online fashion was supposed to make shopping effortless. In reality, opening delivery parcels is often deeply disappointing: over 40% of apparel bought online ends up returned, costing billions in logistics and creating massive environmental waste.

Traditional e-commerce forces shoppers into the "model paradox": a garment looks breathtaking on a 6-foot studio model under staged lights, but size charts and static flat-lay photos fail to answer the single most vital question: "How will this actually drape, fit, and complement ME?"

Furthermore, true style is holistic. It’s not just about fabric silhouette—it’s about complexion harmony, skin undertones, and how accessories like jewelry pull the look together. Traditional virtual try-ons are either clunky 3D avatars requiring specialized hardware, or one-trick image filters that ignore skin tone and accessorizing entirely.

We built Aura TryOn to solve this: a unified, privacy-first virtual fitting room and aesthetic intelligence studio that transforms any casual portrait and clothing/jewelry image into a photorealistic, personalized styling preview—instantly and with zero special hardware.

What it does

Aura TryOn is an intelligent web-based styling studio featuring three integrated modalities:

AI Apparel Try-On (AI Clothes V4 Engine):

Users upload a casual portrait and a garment photo (or select curated presets). Aura’s neural pipeline detects posture, shoulder slope, and torso curvature, seamlessly draping tops, outerwear, bottoms, shoes, or full-body outfits. Preserves natural fabric tension, wrinkles, lapel folds, and environmental lighting for true-to-life visualization. Includes multi-person filtering, garment auto-categorization, and shoe-replacement toggles. 2D Jewelry Virtual Try-On:

Specifically engineered for fine jewelry and accessories. Automatically detects ear anatomy, lobe position, and occlusion, pinning earrings (studs, hoops, chandeliers) with specular metallic luster, accurate gravity drape, and subtle jawline micro-shadows. Skin Intelligence & Complexion Harmony (Powered by Google Gemini AI):

Evaluates facial portraits across dermatological and aesthetic metrics: Fitzpatrick Scale (Types I–VI), Undertone (Warm, Cool, Olive, Neutral), barrier health, and dominant Hex tone estimation. Generates an objective 7-point clinical metric breakdown (hydration, texture, pores, clarity, redness, firmness, dark circles). Dermatology meets fashion: Synthesizes custom fashion color harmony palettes and metal recommendations (e.g., warm gold vs. silver/platinum) that naturally illuminate the user’s specific complexion before they buy. Interactive Comparison, Zoom & Instant Export:

Side-by-side verification and high-resolution downloads so shoppers can share their look or make confident purchasing decisions. Enterprise-Grade Privacy Architecture:

Zero-footprint processing: user photos are kept strictly in ephemeral RAM buffers, never stored on permanent server disks, and automatically purged after a 30-minute TTL.

How we built it

Computer Vision & VTO Core: Orchestrated with Perfect Corp / YouCam AI Clothes V4 (/s2s/v2.0/task/cloth-v4) and 2D Jewelry VTO APIs (/s2s/v2.0/task/2d-vto/earring). Implemented an asynchronous polling state machine and a dedicated same-origin reverse image proxy (/api/jobs/:id/image) to overcome browser CORS limitations on pre-signed S3 storage buckets. Skin Intelligence & Color Analysis: Powered by Google Gemini Vision (gemini-3.5-flash-lite / gemini-3.8-flash) via Google AI Studio. Enforced structured JSON schema outputs for consistent diagnostic scores and wardrobe color recommendations. Incorporated SHA-256 buffer hashing and an in-memory 1-hour cache with exponential backoff to eliminate duplicate API calls and safeguard rate quotas. Media & Story Production: Developed custom headless synthesis scripts using msedge-tts to generate 48kHz natural human studio voiceover and assembled a dynamic 4-minute presentation experience (story.html).

Challenges we ran into

Overcoming Signed S3 CORS Constraints: The upstream VTO engine returns temporary pre-signed S3 bucket URLs upon job completion. Loading these directly in browser elements led to strict cross-origin hotlink failures. We solved this by implementing an in-memory streaming reverse proxy on the Express server that pipes image bytes with proper caching headers directly to the client. Asynchronous Multi-Step Pipeline Synchronization: Each try-on requires registering two discrete image files, obtaining signed upload slots, uploading raw bytes, initiating the neural task, and polling for completion without locking Node.js event loops. We built a robust in-memory job state machine with unique job IDs, polling timeouts, client abort handlers, and automated TTL garbage collection. Balancing LLM Vision Depth with API Rate Limits: Analyzing high-resolution facial images with Gemini can quickly exhaust token quotas or trigger rate limits. We tackled this by engineering a compact, dermatologically structured prompt and deploying a SHA-256 buffer hash cache that recognizes identical uploaded faces and serves immediate cached insights. Natural Garment Draping Across Diverse Postures: Fitting varied garment categories (crop tops vs. trench coats vs. shoes) onto varied candid poses often produced boundary artifacts. We fine-tuned the model parameter mappings (garment_category, change_shoes, and filter_multi_person) to ensure clean segmentations even with background clutter.

Accomplishments that we're proud of

Holistic Fashion Intelligence: Successfully merged generative clothing diffusion with dermatological skin undertone diagnostics into one unified, elegant workspace. True Privacy-by-Design: Achieved a 100% ephemeral in-memory processing pipeline—no customer images are written to disk or stored permanently, giving users complete peace of mind. Fast, Responsive Design: Designed and delivered a custom, high-end dark glassmorphism interface with instant drag-and-drop validation, zero layout shifts, and zero framework bloat. Developer & Demo Experience: Created an interactive developer control center featuring live preset samples, dynamic Gemini model switching, and real-time API health telemetry. Integrated Storytelling: Built and rendered a 4-minute cinematic walkthrough complete with synchronized studio voiceover narration directly into the application

What we learned

Orchestrating Hybrid AI Pipelines: Specialized diffusion models excel at localized warping and texture synthesis, while multimodal LLMs (like Gemini) excel at spatial reasoning, contextual color theory, and structured diagnostics. Combining them unlocks far more value than using either in isolation. Defensive API Engineering: Relying on external AI services requires strict image validation (magic byte inspection), resilient retry strategies, and same-origin proxying to create an unbreakable frontend experience. The Power of Complexion Matching: Color harmony isn't subjective; grounding fashion recommendations in the Fitzpatrick scale and undertone science dramatically increases user confidence in outfit selection.

What's next for Aura TryOn

Full-Outfit Layering & Mix-and-Match: Enabling multi-item try-ons in a single session (e.g., layering a silk shirt under an overcoat paired with tailored trousers and jewelry). 360° & Short-Form Video Try-On: Moving from static portraits to short video clips or multi-angle photos to simulate dynamic fabric drape, movement, and shine. 1-Click E-Commerce Extension & SDK: Packaging Aura as an embeddable widget / Shopify & WooCommerce plugin so online shoppers can click "Try on with Aura" directly on product pages. AI Size & Fit Recommendation Engine: Leveraging Gemini’s spatial analysis combined with brand-specific size charts to suggest exact numerical sizing (e.g., "We recommend Size M based on your shoulder-to-chest ratio"). Encrypted Style Capsule: Allowing users to save their favorite generated looks and skin-matched seasonal color palettes to a private, client-side encrypted lookbook.

Built With

• computer-vision
• css3
• dermatology-ai
• e-commerce
• express.js
• fashion-tech
• ffmpeg
• gemini-api
• generative-ai
• google-ai
• google-gemini
• html5
• image-processing
• javascript
• msedge-tts
• multer
• multimodal-ai
• node.js
• privacy-first
• responsive-design
• rest-api
• text-to-speech
• virtual-try-on
• youcam-api