OnMe

That outfit you just saw? Here it is on you, and where to buy it in your colours.

Comment

Inspiration

The gap between "that looks good" and "that looks good on me".

Product photography is shot on a model who is not you, under lighting that is not yours. Of everything you cannot judge from a flat-lay, the hardest is whether a colour will read against your own skin or vanish into it. The existing answer to that question is a colour-season quiz — a vibe with a vocabulary attached, and no measurement anywhere in it.

Perfect Corp already owns the hard primitive: measurement of the person. What nobody had done was turn that measurement into a purchase filter — a thing that answers "should I buy this?" rather than "how do I look in this one brand's catalogue?"

What it does

Point OnMe at any garment — in a feed, on a shop page, anywhere — and it does three things in one pass:

  1. Puts it on you. A real YouCam Apparel Try-On (cloth-v4) render of that garment on the demo user.
  2. Tells you whether the colour works. Your skin is measured once with the YouCam Facial Color Tones Analyzer; every garment is judged against that one reading by CIEDE2000 colour difference, against a stated bar. The answer is works on you or disappears on you — in those words, with two swatches side by side so you can see it yourself.
  3. Finds where to buy it. Live web search, real merchants, prices shown only where a merchant stated one.

Three surfaces, one answer: a web app anyone can open, a Chrome extension that right-clicks any outfit photo, and an Android overlay — a floating bubble you summon over any app, where you drag a box around a garment and get it back on yourself. All three land on the same verdict page, so what the phone shows is exactly what a judge opens.

Two things it does that a quiz cannot:

  • It refuses. Point it at a stripe or a print and it says "this is a pattern, not one colour — there is no single colour to judge." That refusal is computed from the spread of the sampled pixels against a stated bar, not guessed. It still shows you the garment on.
  • It shows the near-miss. A tan sweater against this profile is almost your exact tone — and in the render you can watch it disappear. A tool that only ever delivers good news is not measuring anything.

How we built it

  • Web: Next.js 16 (App Router) on Vercel. The verdict page is one column: the render first, the reading under it, the working one tap below that.
  • Colour engine: pure TypeScript — sRGB→CIELAB, CIEDE2000, ISCC–NBS Level-3 centroid naming. No dependency does the maths.
  • YouCam APIs: Facial Color Tones for the one skin measurement, cloth-v4 for every try-on, text-to-image for the synthetic demo user and garment photography.
  • Shopping: live web search behind a ShopSearchProvider interface, captured once and replayed so the demo is deterministic.
  • Extension: MV3, a context-menu picker and a shadow-DOM sheet — a shadow root rather than an iframe, because the frame-src policy of real shopping sites blocks the iframe and cannot touch the shadow root.
  • Android: Kotlin, SYSTEM_ALERT_WINDOW bubble plus MediaProjection capture and a drag-crop.
  • The corpus is content-addressed. A result id is derived from the garment bytes, so the same garment always resolves to the same page — which means the demo corpus can be committed as files and /r/<id> works on a cold lambda, offline, a year from now, with no database to keep alive.

Challenges we ran into

A paid, tens-of-seconds render does not fit inside a request that has to answer immediately. Putting cloth-v4 inline broke three ways at once: the Android client gives up at a 30-second read timeout, the serverless function has its own shorter ceiling, and worst — a function killed mid-poll abandons a task that has already been charged and whose result is unrecoverable. You pay for an image nobody receives. The fix was to stop treating the render as part of the request: the verdict endpoint returns a deterministic image URL, and the render lives behind GET /api/tryon/<key>, which resolves in order of cost — committed file, then cache, then a live call cached on the way out. The page then contains nothing more exotic than <img src="/api/tryon/…">. Measured cold: 14.9 s, 28.3 s, 45.0 s. Cached: 0.22 s.

Three defects only a real device could surface. An activity written but never declared in the manifest (fatal on the first bubble tap). A bubble that was a single near-black disc, invisible on a dark wallpaper. And repeat taps doing nothing, because acquireLatestImage() returns null unless a new frame exists and a still screen produces none — so hiding the bubble is now itself the trigger, since that is a change.

The measurement was right and the sentence about it was wrong. Twice. A tilted solid garment was being called "a pattern" when the real problem was that the crop box held too much background — a claim about the clothes when the fault was the framing. And a refused garment reported "no shops came back", describing a search that never happened, because the query is built from the garment's colour and the refusal is precisely the statement that there isn't one.

Accomplishments that we're proud of

  • Every verdict carries its own error bars. The skin and garment points are each perturbed by a declared radius; if the answer flips anywhere inside that envelope it reports borderline instead of rounding to a clean yes or no.
  • CIEDE2000 is verified against the reference data — the 34 pairs of Sharma, Wu & Dalal (2005) Table 1, asserted to four decimal places. That is the conformance set the formula's own authors published for implementations to test against.
  • No language model sits anywhere in the verdict. It is colour science and a stated threshold, end to end.

What we learned

  • Precision in the measurement vocabulary is imprecision in the shopping one. The same garment searched as its ISCC–NBS centroid name ("strong yellowish green") returns yarn wholesalers; as its specific retail name ("Greenish Khaki") returns fabric suppliers; as its colour family ("Khaki green") returns real retailers with prices. The code carries all three names and uses each only where it belongs.
  • A number is not an explanation. Two colour swatches side by side outperform the distance between them, because a person can see "those are the same" without being taught what the number means.
  • The absence of a confidence value is itself a finding. The analyzer returns a skin colour and four quality grades but nothing about how sure it is. For a product whose whole claim is measurement, that gap had to become a stated assumption rather than an invisible one.
  • A provenance chip has to describe what the reader is looking at, not what the author once did. A committed fixture that was captured live must say cached, because the person reading it is replaying, not fetching.

What's next for OnMe

  • Fit. The largest unserved axis in the purchase decision and a genuinely different technical problem from colour, not an extension of it.
  • A portable profile. A measurement that dies at the edge of one storefront is worth far less than one measured once and applied everywhere — which likely means the profile has to be held on the consumer's side to be portable at all. That is a trust question before it is an engineering one.
  • Assistive use. For a shopper with colour-vision deficiency, a measured verdict is not a convenience, it is access.
  • A wider corpus, and a second measured profile, so the same garment can be shown answering differently for two different people — which is the whole thesis in one screen.

Built With

Share this project:

Updates

Submission history