SkinSense Agent

Inspiration

Skincare shopping is a wall of confident claims and no measurements. People buy for the skin they wish they had, guess at results in the mirror, and abandon routines that were never matched to their actual concerns. We wanted the shopping loop to start from a measurement instead of a marketing claim — and to keep listening after the first answer, the way a good human consultant does.

What it does

SkinSense Agent runs a four-step agentic loop:

1. Describe — the user states a concern in plain words ("dry and tight by midday, cheeks flush red, dull patches") and can add a front-facing selfie.
2. Analyze — the app calls the YouCam Skin AI Analysis API (File API upload → skin-analysis task → polling) and scores eight concerns: moisture, oiliness, redness, acne, pores, texture, wrinkles, radiance — plus an overall score and estimated skin age.
3. Recommend — the agent ranks the concerns weakest-first and builds an AM/PM routine from a product catalog: every card shows the product, a one-line pitch, the agent's reason ("aimed at Redness, your weakest area"), and its price; tinted finishers carry a virtual try-on hook via YouCam Makeup VTO, honestly labeled as a placeholder in this build (the VTO API requires the photo at a publicly accessible URL, and SkinSense never publishes a user's face). The basket total is computed honestly — products shared between AM and PM are counted once.
4. Refine — the user pushes back in plain language ("too expensive — under $20 each", "everything fragrance-free, my skin is sensitive", "simpler", "no makeup") and the agent rebuilds the routine under the new constraints, listing exactly what changed: swaps by name, constraint notes, and the before/after basket total. The live demo was verified against the real API on October 23, 2026: real scores on the author's own selfie (overall 74, skin age 31; weakest: redness 55, pores 66), a routine built from them ($150 basket), and a filmed refine round ("under $20 each and fragrance-free") that named 7 changes and cut the basket to $66.

How we built it

• Zero-dependency Python (standard library only) — a small threaded HTTP server keeps the API key server-side; the browser never talks to Perfect Corp directly.
• Real API integration, documented spec — youcam.py implements the official flow from Perfect Corp's docs: POST /s2s/v2.0/file/skin-analysis for a pre-signed upload URL + file_id, a PUT of the image bytes, POST /s2s/v2.0/task/skin-analysis with the SD dst_actions set and format "json", then polling GET /s2s/v2.0/task/skin-analysis/{task_id} and normalizing data.results.output.
• Deterministic agent — the routine builder and feedback parser are transparent rules over the scores (positional concern weights, constraint filters, product-level diffing), so behavior is testable and identical in demos; no LLM bill, no nondeterminism.
• Honesty as a feature — when no API key is configured (or no photo is provided), the app runs on deterministic, keyword-aware SAMPLE data and labels it in the UI badge, an amber banner, and the analysis source line. Sample output is never dressed up as a real measurement.
• Design — a single-page, framework-free UI with a warm editorial look (serif display type, soft cards, animated score bars with the three priority concerns flagged) built to be judged as a consumer product, not a developer console.

Challenges we ran into

• The API is asynchronous with a two-step file upload (metadata → pre-signed PUT); the docs warn that skipping the PUT surfaces later as opaque 500/404 errors, so the client validates each stage and fails with the API's own error text.
• SD and HD concern sets cannot be mixed in one task (InvalidParameters), so the client pins one documented SD set.
• The engine rejected a full-frame selfie as "face too small"; a tighter crop of the same photo was accepted — the demo films the crop, and the submission notes disclose it.
• Keeping the demo truthful: the fallback path had to be as designed as the live path, because reviewers may run the repo without a key.

Accomplishments that we're proud of

• Full loop working end to end today, verified against the real API (Oct 23, 2026): real scores on the author's own selfie, a routine built from them, and a filmed refine round, all in the demo video.
• Next: complete the live Makeup VTO wiring in a privacy-preserving way, swap the demo catalog for a retailer's real product feed, and add week-over-week score tracking so the agent can answer "is this routine working?"
• Value: for consumers, a routine derived from measured concerns instead of influencer claims, with a budget ceiling the agent actually respects. For retailers, an agentic storefront pattern — measure → recommend → try on → refine — that raises conversion quality and cuts returns. The refine loop is the differentiator: most "AI skin" demos end at the score screen; this one keeps negotiating.

Built With

• ai
• analysis
• api
• css
• html
• javascript
• makeup
• python
• skin
• vto
• youcam