Mirror

Your skin colour picks the shade. Your skin condition picks the formula. One selfie, twelve YouCam endpoints, every product naming the measurement that chose it.

Comment

Inspiration

Every shade finder online does half the job. It matches a colour, hands you a list, and stops. Nothing you can measure about your face — oiliness, dryness, redness, texture, face shape — enters the decision.

So the right colour arrives in the wrong formula. A matte foundation on dry skin looks like chalk; a dewy one on oily skin slides off by noon. Both were the correct shade. Both get returned or abandoned in a drawer.

That gap is the whole project. Skin has two independent properties, and shops only ever use one:

  • Skin colour picks the shade.
  • Skin condition picks the formula.

Two people can match the identical foundation shade and need opposite products. The diagnosis changes the product, not just the caption.

What it does

One selfie, then five things:

  1. Scan — three YouCam reads run in parallel off a single upload: skin colour, seven skin concerns, and face attributes.
  2. Diagnose — colour becomes an undertone, a depth and a seasonal palette. Concerns become a formula: finish, glow, coverage, under-eye correction. Every line prints the reading that produced it: "Matte finish, because your skin reads oily (78%)."
  3. Shop — seven aisles, each ranked by a different question, because they're asking different things. Foundation must match your skin. Blush and clothes must flatter it. Skincare aims at your worst measured concern.
  4. Try on — foundation, lipstick, blush, clothes, hair and whole looks render on your own photo, before and after, side by side. The makeup render carries your prescribed intensities, so it's your formula in that shade rather than a stock preset.
  5. Buy — anything a listed store sells can be ordered, and the store sees it in their dashboard.

And on the other side of the counter: a shop signs in, uploads a spreadsheet or points at a feed it already publishes, and its products rank against real shoppers within a minute. No hand-tagging — colours are measured from the product photos server-side. Or it skips Mirror entirely and drops the engine onto its own storefront with one <script> tag.

How we built it

Bun and TypeScript on the server, React on the front, a single proxy so the YouCam key never reaches the browser.

Thirteen YouCam endpoints, and nothing is called for display only — every response changes a product that gets recommended:

Endpoint What it decides
skin-tone-analysis The spine. Skin hex → undertone → palette → ranks every aisle
skin-analysis Seven concerns → the formula. The half other tools skip
face-attr-analysis Face shape ranks hairstyles; gender orders shelves
makeup-vto Foundation, lipstick, blush at your prescribed intensities
cloth-v4 Any garment from its own product photo
cloth YouCam's template garments (v4 dropped template_id)
hair-transfer A cut, rendered
look-vto A whole look in one pass
skin-simulation Your face after a course of treatment
template/* ×3 YouCam's garment, look and hair catalogues
file One upload, reused by all nine tasks

Two decisions worth defending:

Undertone is not hue. Hue is correct for product colours across the whole wheel, but measured on real skin it saturates at 0.94–0.99 for everything — every shopper comes out orange. Yellowness over redness in CIELAB separates cleanly: golden 3.1, olive 11.0 against pink 1.1, rosy 0.7. That ratio decides it.

The language model is a re-ranker, never a retriever. Code narrows each aisle by colour distance; the model picks one product from a shortlist of twelve and writes one sentence. Every id it returns is verified against the shortlist it was offered — not merely against the catalogue, because a real product it was never shown is also wrong. Without a key the app still recommends, labelled as a colour match rather than advice.

Challenges we ran into

The scores are inverted, and it's silent. YouCam scores health: a low oiliness score means oily skin. Invert it twice and you prescribe the exact opposite formula — matte for dry skin — and it still reads completely plausible. We invert once, in one function, with a test that feeds live API payloads verbatim.

Real scores cluster at the top. Our first calibration used absolute thresholds, and on a well-lit face every concern comes back 90–100. The thresholds never fired and every shopper got identical advice. Undertone is now decided by comparing redness against that person's own mean, so the reading responds to their actual variation.

Nail polish remover, recommended for acne. We were assigning what a product treats from the category it was filed under, so every row in "cleansers" claimed acne. Testing severe acne surfaced Dissolvant pour les ongles at the top — worse than recommending nothing. What a product treats now comes from what its name evidences, with the category only as a fallback.

Twelve near-identical whites. One store's neutral-heavy catalogue took the entire clothes shelf on straight palette distance. Both colour rankers now cap each brand and top up, the same way the foundation ranker already handled dense shade ranges.

API traps that cost real hours. skin-analysis defaults to a ZIP with a completely different response shape. skin-tone-analysis returns results as an object, so results[0] yields nothing and looks like it works. face-attr-analysis rejects the documented nested envelope and wants a flat body. Uploads expire and then fail as unknown_internal_error, which says nothing — we map it to "scan again". They're all documented in the README so the next person doesn't pay twice.

What we learned

Depth of API use isn't endpoint count — it's whether each response changes a decision. The endpoint that surprised us most was skin-simulation: we'd written off skincare try-on as impossible, since a serum has no visible application. But the outcome is exactly what a shopper is buying, and rendering it on her own face, scaled by her own concern severity, is a stronger argument than any product copy.

We also learned to distrust our own plausible output. Every bug above produced results that looked fine. The ones we caught, we caught by testing against real payloads and real skin tones rather than reading the code back.

What's next

  • Fitzpatrick typing to deepen the diagnosis beyond depth buckets
  • Nail try-on (nail-vto is wired and verified; it needs a hand-photo flow)
  • Return-rate measurement — the real proof is whether a prescribed formula gets returned less often than a matched shade alone

Built With

Share this project:

Updates