Inspiration
Around 300 million people have color vision deficiency. More than half are mild deuteranomals who were never diagnosed, because their variant causes no darkening: they simply grew up being told they were bad at colors.
One sentence reorganized everything for us:
A person with color vision deficiency cannot see the color of their own face, and therefore cannot make any decision that depends on it.
Not correcting redness. Not knowing what suits them. Not telling a navy sweater from a dark purple one in their own wardrobe. That last one sounds small until you learn what people actually do about it: they ask someone. Every morning a partner, a parent or a roommate tells them whether the outfit works. And when they shop, they buy black, gray, navy and white — the safe colors — or a third navy shirt they cannot tell apart from the two they already own.
YouCam's skin-tone-analysis returns the color of their face as a number. There, and only there, the API does something no amount of local code can: it substitutes for a missing sensory channel. That is not a beauty use case. It is assistive technology for shopping.
Then came the corollary that governed every decision: our user is the one person who cannot audit our product. If we name a color wrong, they have no way to catch it. So honesty could not be a promise in a README. It had to be structural.
What it does
KROMA measures how you see color, measures the colors you cannot see, and then answers the shopping and dressing questions that depend on both — stating how confident it is, and refusing when the light will not let it know.
It measures the user first, without asking. A Farnsworth D-15-style arrangement plus three adaptive staircases estimate the type and severity of the deficiency. We never ask which one you have: more than half of our users do not know, and asking would be asking for the answer they came for. The confusion axes are derived from the same Machado 2009 matrices the app later uses to simulate your vision.
Skin AI — it measures the face. Skin, eyes, brows, lips and hair, plus the Fitzpatrick phototype and redness.raw_score. We use raw_score, never ui_score, because the documentation says ui_score is adjusted upward as a psychological motivator — and our user could never detect us flattering them. Redness is not interpretable without the phototype, which makes the Fitzpatrick × redness pairing a technical necessity, not a list item. Hair is labeled as what the engine returns: one of seven preset categories, not a measurement.
Skin AI — it checks its own advice. From the measured skin, KROMA computes the color corrector that cancels the redness — the skin's own color with its red axis mirrored, at the same lightness and warmth:
$$ \text{skin } (L^{\ast}, a^{\ast}, b^{\ast}) \quad \longrightarrow \quad \text{corrector } (L^{\ast}, -a^{\ast}, b^{\ast}) $$
Then makeup-vto renders it onto the user's face and the redness is measured again on the same image — including when it gets worse. On already-calm skin we measured 72.5 → 68.9 (worse; higher is calmer); on skin with real redness, 57.0 → 58.4 (better). We show both, because a product that can tell you not to buy a corrector is worth more than one that always flatters.
eCommerce VTO — "Should I buy this?" Photograph a garment in a shop and KROMA answers before you pay: its measured color and name, whether it collides with your face, which of your clothes it goes with — a percentage that counts only safe pairs — and a warning if you already own one you could not tell apart.
eCommerce VTO — on your body. cloth-v4 renders the garment onto your own photo (with outer for coats and the multi-person filter on, so only you get dressed), scarves and hats frame the face, and whole outfits are built by feeding each render back in as the body for the next piece. The picture answers how does it hang; the color verdict next to it comes from the measurement, and the screen says so.
It answers the morning question. Get dressed composes complete outfits with no clashing pair, judged against the palette on your own face, and says what the product exists to say: you can put any of these on without asking anyone. You can also build and check your own — many people already wear outfits someone they trust put together, and nothing has ever been able to check those.
It makes the invisible visible. For a color-blind user, the original image is already their view — their eyes do the simulation. So instead of showing them a simulation, KROMA moves the information they lose into channels they still receive (Fidaner, Lin & Ozgüven, 2005):
$$ \mathbf{c}' = \mathbf{c} + k\,M\,(\mathbf{c} - S_s\,\mathbf{c}), \qquad k = \text{strength} \times \text{severity} $$
where \( S_s \) is the Machado 2009 matrix for the user's measured severity. It is a presentation filter only: it is never measured, never sent to YouCam, and the build fails if an API route imports it.
How we built it
Stack. Next.js 15 (App Router), TypeScript strict, Tailwind v4, IndexedDB via idb, culori for CIELAB and CIEDE2000, sharp on the server, deployed on Vercel. A mobile-first PWA, installable on Android, iOS and desktop.
No database and no accounts — on purpose. A login means a judge creates an account and lands in an empty app anyway, because the wardrobe is theirs. Everything lives in IndexedDB on the device. The app starts empty: every color on screen was measured from the user's own photo.
Ten YouCam endpoints across six server routes. Every call is server-side; the key never reaches the browser.
| Route | YouCam endpoint | What it contributes |
|---|---|---|
/api/portrait |
skin-tone-analysis ×2, lighting, fitzpatrick-scale-analyzer, v2.1/skin-analysis |
face colors raw and light-normalized (the gap between them is the confidence), phototype, redness |
/api/garment |
sod |
isolates the garment from the surface it lies on |
/api/tryon |
cloth-v4 (MCP first, REST underneath), scarf, hat |
the garment on the user's photo |
/api/outfit |
the same, chained | the render becomes the body for the next piece |
/api/makeup |
makeup-vto + skin-analysis |
renders the corrector, re-measures the redness |
/api/mcp |
lighting, face color and phototype over MCP | an audited run showing which transport answered each step |
Plus the File API upload — and the PUT to the pre-signed URL everyone forgets.
Integration details that decide whether a selfie is accepted. The color analyzer refuses a head turned more than 10° by default; we send face_angle_strictness_level explicitly and retry once at 15°, labeling the result approximate. The skin analyzer wants the face over 60% of the image width; the makeup check retries on a tighter crop — cropping selects pixels, it cannot move a color. Every image is converted server-side (EXIF rotation, flattened onto white, 4:4:4 chroma JPEG) into a size window each endpoint accepts. Every request runs on a time budget below Vercel's 60 s ceiling, and a 429 is retried with backoff before the user is told the service is busy.
Model Context Protocol. Official SDK over Streamable HTTP, the same key as REST, sessions opened server-side. The fashion server renders every clothes try-on in production; beauty and creators run in an audited flow. Tool names are resolved against each server's own tools/list, and a failure written inside a successful MCP response still counts as a failure.
Where the API is deliberately not used — the other half of the architecture: a garment's color comes from our own pixels, its name from a deterministic derivation, the vision simulation from published Machado 2009 matrices (297 coefficients, transcribed mechanically and sealed with a SHA-256), and every verdict from local arithmetic. No language model decides anything — not a color, a name or a verdict. If a model said "olive green" about a brown garment, anyone else would catch it instantly. Our user never would.
The verdict, in one rule. Two colors clash for this user when their hues collide in the user's simulated space with lightness equalized, \( \Delta E_{00}^{\text{hue}} < 11 \) — unless lightness separates them anyway (WCAG contrast \( \ge 4.5 \) is safe, \( \ge 1.8 \) is risky but distinguishable). A measurement taken in poor light can never produce a confident "safe".
Challenges we ran into
1. The lighting pass normalizes faces — and damages garments. Our confidence layer compares a raw reading against a light-normalized one. On a face that works: the observed gap is ΔE 0.5–0.7, a real color cast removed. On a garment, measured on the live endpoint under five lights, the gap was unrelated to the truth: it would have refused the best photo in the set (ΔE 16.9) and accepted one with strong folds. On that daylight shot the normalization moved the garment from ΔE 1.8 to ΔE 20.2 away from its true color. The engine relights scenes; flat fabric is not a scene. So lighting runs on faces only, and garment confidence is built from our own pixels: clipping, exposure, the 8-bit encoding floor and framing.
2. Shading is not a second color. Grouping fabric pixels naively turned a plain orange T-shirt into "37% dominant — patterned": folds, not a print. A fold is a continuous ramp; a print is a gap. Grouping by hue, pooling the grays, and splitting only at a real valley in the lightness histogram reads the same shirt as 99% dominant, solid.
3. A rule that could never fire. Our key rule says a pair colliding in hue but separated by lightness is not a clash. Measured on full CIEDE2000, the two conditions were mutually exclusive: across 130,816 pairs covering sRGB, 8,147 collided and 0 were ever rescued. Splitting the axes — hue with lightness equalized for collision, WCAG contrast for rescue — made the rule reachable.
4. The two transports disagree. Same image, same file_id: skin agreed within ΔE 1.0, the iris differed by ΔE 6.4. A user routed one way would be told a different eye color than a user routed the other, and could never detect it. Our policy: REST owns every measurement that feeds a verdict; MCP renders and orchestrates.
5. Selfies refused for reasons the user cannot see. Three engines, three framing vocabularies, and the strictest rule in the API on the skin analyzer. Every code the documentation lists maps to a sentence telling the user what to change — including how many degrees to straighten up.
Accomplishments that we're proud of
- Honesty implemented rather than promised.
refuseis a feature with its own screen.nullnever renders as0, because "not measured" and "measured, perfect" are different claims. Every verdict carries the confidence of the measurements behind it. - Measurements that overturned our own design — the lighting pass, the shading model and the collision metric each changed once the evidence came in, and the evidence is published.
- A skin recommendation that checks itself: compute a corrector from measured skin, render it, re-measure the redness, report the delta — including when it worsens.
- Calibration verified against synthetic observers. A harness feeds the scorer complete sessions from model observers — normal; deutan at 0.4, 0.6 and 1.0; protan at 0.6 and 1.0; tritan at 0.8 — and recovers every type, while a random guesser and an always-wrong responder are refused.
- An interface that works for someone who receives no color at all. Every state travels as icon + text + shape. Even in the color-vision test, where the stimulus has to be color, every swatch is numbered.
What we learned
- Read
tools/listand the OpenAPI specs even when you call REST. They carry complete enums and tolerances —outer,filter_multi_person, the fiveface_angle_strictness_levelvalues — and both are free. - Follow
next_token. The price list is paginated, and its first page looks complete while containing none of the Skin AI or try-on SKUs. - Checking is not trusting a flag.
isErrorcan be absent on calls that failed; read the payload. - Test your instrument in both directions. It is trivial to "fix" a false refusal by refusing nothing, or a false pattern alarm by calling everything solid — which would blind the exact person the product is for.
- Physics is not a gap in effort. Without a surface of known color in the frame, the illuminant cannot be recovered. We do not claim to detect a color cast on a garment photo.
What's next for KROMA
- Guided capture with the JS Camera Kit's
shadefinderandskincaremodes, which check framing, angle and light in real time — the most common reasons a selfie is refused. - Webhooks instead of polling, removing the 60 s ceiling so an outfit can render with more than two pieces.
- Foundation on the face:
makeup-vtohas afoundationeffect, and KROMA already computes the shade. - Brettel 1997 for tritanopia, where Machado 2009's own source says it is not accurate.
- A browser extension for shopping sites: product photos are public URLs, every task accepts
src_file_url, and a garment could be judged against your face and wardrobe without leaving the store. - Retail integration: a catalog filtered by "show me only what I can tell apart from what I already own."
Built With
- accessibility
- ai-skin-analysis
- background-removal
- ciede2000
- cielab
- color-vision-deficiency
- culori
- fitzpatrick-skin-type
- github-actions
- indexeddb
- makeup-virtual-try-on
- mcp
- model-context-protocol
- next.js
- node.js
- perfect-corp
- pwa
- react
- sharp
- skin-tone-analysis
- tailwind-css
- typescript
- vercel
- virtual-try-on
- youcam-api