"ClimaSkin VTO

Fit the climate. Match your skin. Wear with confidence.

Comment

Inspiration

E-commerce fashion has made global apparel accessible at our fingertips, yet it remains fundamentally disconnected from two critical human realities: our individual skin biology and our rapidly changing local environment.

While working on climate resilience and sustainable systems, we observed two persistent inefficiencies:

High e-commerce return rates—exceeding 25–30% globally—driven primarily by poor fit and jarring color mismatches against the buyer's natural skin tone. This creates millions of tons of landfill waste through reverse logistics.

  1. The hidden impact of microclimates on skin health. Extreme heat, rising UV indices, and soaring humidity lead to severe dermal irritation (e.g., eczema, rosacea, heat rashes) when paired with non-breathable synthetic fabrics or harsh chemical dyes.

  2. We built ClimaSkin VTO to bridge this gap. We wanted to move beyond superficial visual try-ons by creating a multi-modal assistant that treats apparel as an active layer of microclimate defense—ensuring clothing looks stunning, feels comfortable, and protects the wearer's skin.

    What it does

  3. Analyzes Your Skin Health & Tone Uses the YouCam Skin AI API to scan a selfie and extract quantitative dermal metrics:

Skin Undertone (Cool, Warm, Neutral) Redness & Sensitivity Index Hydration & Barrier Condition

  1. Reads Real-Time Local Microclimate Data Ingests ambient environmental parameters based on your location: UV Index Temperature & Relative Humidity Air Quality & Pollution Levels

  2. Filters & Scores Apparel for Safety and Aesthetics Applies a multi-criteria scoring algorithm to candidate clothing:

Color Harmony: Matches your skin undertone to flattering garment color palettes.

Derm-Safe Fabrics: Flags abrasive synthetic fabrics or heavy dyes if high skin redness/sensitivity is detected, recommending breathable, hypoallergenic options (e.g., organic linen, silk, UPF-rated cotton).

  1. Renders Real-Time Virtual Try-On (VTO) Uses the YouCam Apparel VTO API to overlay selected garments onto your portrait with realistic lighting, texture, and drape alignment.

  2. Delivers Actionable Outfit & Care Advice Displays the final virtual try-on alongside:

A Fabric Safety & Comfort Score

Dermal Warnings for extreme weather conditions

Skincare Prep Recommendations (e.g., SPF or barrier cream pairings) to complement your chosen outfit.

How we built it

Dermal Diagnostic Engine: Built around the YouCam Skin AI API, which analyzes high-resolution user selfies to extract quantitative metrics such as skin undertone (Cool, Warm, Neutral), redness, hydration, and overall sensitivity.Climate Context Service: Ingests real-time ambient parameters—ambient temperature ($T$), relative humidity ($RH$), and UV index ($UV$)—based on the user's location coordinates.Harmonic & Fabric Scoring Model: A decision matrix that filters candidate catalog items based on material breathability, UPF rating, and color harmony.Visual Try-On Pipeline: Passes the user's base photo and the filtered garment asset URL to the YouCam Apparel VTO API, generating realistic garment drape, texture, and lighting alignment.

Challenges we ran into

API Latency & Asynchronous Pipelines: Chaining skin analysis, weather lookup, and virtual try-on image generation introduced high end-to-end latency. We overcame this by parallelizing the YouCam Skin AI API call and the local climate data fetch, reducing total pipeline waiting time by over 40%.Lighting and Undertone Calibration: Natural light variations significantly skew undertone detection. Normalizing input photo color temperature prior to sending payloads to the Skin AI API was essential for maintaining consistent color matching scores ($C_{\text{undertone}}$).Fabric & Dermal Rules Mapping: Translating qualitative skin conditions (e.g., "high redness / low hydration") into quantitative fabric recommendations required building a domain-specific lookup table that categorizes textile weaves by moisture-wicking and friction thresholds.

Accomplishments that we're proud of

Seamless Multi-Modal API Integration: Successfully chained the YouCam Skin AI API, YouCam Apparel VTO API, and real-time environmental weather services into a unified, low-latency microservice pipeline.Pioneering "Derm-Safe" Fashion: Built a custom multi-criteria scoring algorithm ($S_{\text{fabric}}$) that quantitatively pairs skin barrier health (redness, hydration, sensitivity) and undertone harmony with fabric breathability and UPF protection.Parallel Processing Optimization: Overcame high end-to-end network latency by parallelizing diagnostic visual scans and weather telemetry requests, reducing total processing time by over 40% for a smooth user experience.Climate-Resilient & Sustainable Commerce: Created a practical, high-impact retail concept that directly addresses fashion’s high return rates and reverse-logistics waste by ensuring consumers pick garments that match their natural undertone and skin sensitivities on the first try.End-to-End User Experience: Delivered an intuitive interface that goes beyond aesthetic clothing previews to give users actionable dermal care warnings, climate readiness scores, and skin-prep advice alongside their virtual try-on.

What we learned

Diagnostic Precision Enhances VTO: Virtual try-on becomes exponentially more useful when paired with personal health metrics. Users care as much about how a garment feels on their skin in $32^\circ\text{C}$ heat as they do about how it looks.API Composition Power: Combining specialized APIs—specifically pairing YouCam's visual AI models with third-party environmental telemetry—unlocks entirely new categories of climate-adaptive retail tools.Circular Impact of Fit: Preventing returns at the point of decision by resolving color mismatch and skin discomfort directly reduces carbon expenditure in e-commerce reverse logistics.

What's next for "ClimaSkin VTO

  1. Advanced Fabric-to-Dermal Sensor IntegrationIoT Weather & Sweat Telemetry: Integrate with wearable sensors (e.g., Apple HealthKit, Garmin, Smart Watches) to ingest real-time biometric metrics like body surface temperature, galvanic skin response (sweat rate), and local UV exposure instead of relying solely on static weather APIs.Micro-Weave Structural Modeling: Expand the fabric diagnostic engine by partnering with textile databases to evaluate microscopic thread density, air permeability ($m^3/m^2/s$), and specific chemical dye toxicity levels against sensitive skin conditions.2. Full-Body VTO & Dynamic Movement Simulation3D Multi-Angle & Video VTO: Upgrade from single-photo overlay to 360° dynamic video rendering powered by Perfect Corp's generative video APIs, allowing users to observe how garment drape and fabric tension change while walking or moving.Layering & Multi-Garment Fitting: Enable multi-piece virtual try-ons (e.g., base layers, insulating coats, UV outer shells) that assess combined thermal retention and total skin breathability.

  2. Retail & E-Commerce Integration (B2B Expansion) Shopify & Magento Merchant Plugins: Package ClimaSkin VTO into a plug-and-play e-commerce widget so online retailers can offer skin-smart size/color recommendations directly on product pages.

Returns Reduction Analytics Dashboard: Provide retail merchants with aggregated insights showing how undertone matching and fabric sensitivity filters reduce product return rates and lower reverse-logistics carbon footprints.

  1. Climate Risk & Disaster Relief Adaptation Emergency Response & Field Workwear Edition: Adapt the core engine for disaster relief agencies, outdoor field researchers, and agricultural workers to rapidly assign high-UPF, moisture-wicking, and flame-retardant field gear matched to worker skin barrier health.

Extreme Heat Warning Alerts: Send push notifications when regional climate conditions exceed safe skin exposure thresholds, recommending appropriate protective clothing layers before users step outdoors.

Built With

  • beauty-tech
  • climaskin-vto
  • climate-resilience
  • climate-tech
  • computer-vision
  • dermal-diagnostics
  • e-commerce-ai
  • fashion-tech
  • fastapi
  • machine-learning
  • nextjs
  • open-weather-api
  • python
  • react
  • rest-api
  • retail-innovation
  • sustainable-fashion
  • tailwindcss
  • typescript
  • under-tone-matching
Share this project:

Updates