Inspiration
Online beauty advice says to copy a creator's shade. But nobody sees a shade alone. They see it against the skin underneath. The same lip colour changes impression and harmony on different skin (Seung et al. 2020), and impressions follow the colour difference between lip and skin (Wu et al. 2024). AI makeup transfer also shows its worst artefacts on Black-presenting faces (Al-Otaibi et al. 2026), and foundation ranges skew to light and medium tones (Frisby 2019).
So we changed the question. Not "which lipstick did the creator wear?" but "what did that lipstick do on the creator's face, and which real product does the same on yours?"
What it does
Mirette is a phone-first makeup website. The name comes from the French se mirer, to see yourself in water, so every screen is a photo above moving black water.
- Guided selfie: takes front, left and right photos by itself, with live coaching on pose and light.
- Face reading: YouCam reads colour, skin and shape, and Mirette turns them into a guide to what this face needs.
- Fit: every product gets its best shade, a fit score and a plain reason.
- Try-on: shades and whole looks appear on your own face.
- Look translation: a creator's look is rebuilt from real shades for your skin.
- Shopping: compare sellers, find a near-identical colour for less, and see how many foundation shades really reach your skin.
Mirette scores how a product fits a face. It never scores the face.
How we built it
Stack: Next.js 16, React 19, TypeScript, Neo4j 5, MediaPipe Tasks Vision, raw WebGL2, sharp, and the YouCam API.
YouCam API
Each call follows the same flow: register the file, upload it, create a task, then poll for the result.
- Facial Color Tones (20 units): skin, lip, eye, brow and hair colours, the input to all colour maths.
- Skin Analysis (14 units): 11 concerns, including oiliness, moisture, texture, pores, redness and dark circles. These drive formula matching.
- Face Attributes (20 units): eyelid, eye shape, lip shape, face shape and cheekbones. These drive physical fit and creator matching.
- Makeup Virtual Try-On (1 unit per render): the real render on the shopper's face.
A full scan costs 54 units, so a budget guard protects the free allowance. MediaPipe gives an instant preview on the device, and the YouCam render then replaces it.
1. Guided capture
Head turn is estimated from face landmarks, using the nose and the two cheek edges:
$$\mathrm{yaw} = \frac{x_{\mathrm{nose}} - \frac{1}{2}(x_{\mathrm{left}} + x_{\mathrm{right}})}{\left| x_{\mathrm{right}} - x_{\mathrm{left}} \right|}$$
A photo is taken when the face stays in position for 750 ms, with a yaw of at least 0.12 for each side. The app also checks brightness, distance, centring and tilt. The front photo goes to YouCam. The side photos stay on the device and flag uneven light when a side reading differs from the front by more than a CIEDE2000 of 9.
2. Skin tone
All colour work happens in CIELAB. Depth uses the Individual Typology Angle rather than Fitzpatrick types (Ulrich et al. 2025):
$$\mathrm{ITA} = \frac{180}{\pi}\,\mathrm{atan2}(L - 50,\ b)$$
3. Colour on the face
Makeup sits as separate particles over partly exposed skin, not as a solid film. Following Park et al. (2026), we use a smoothstep interpolation with a small lightness-dependent exponent. For a base colour B, product colour P and coverage c:
$$\hat{c} = c^2(3 - 2c)$$
$$\lambda = \max\big(-40,\ \min(40,\ L_P - L_B)\big)$$
$$\omega = \hat{c}^{\ 1 - 0.006\lambda}$$
$$F = B + \omega\,(P - B)$$
B is the skin for face products and the natural lip for lip products. The form is from Park et al., who reported a mean CIEDE2000 of 3.01 under leave-one-subject-out validation, on foundation measured on forearms. The constants are our own estimates, and using the form on lips and cheeks is our extension.
4. Optical and physical fit
Foundation is scored by the on-face gap and the direct gap to the skin, clamped to the range 0 to 1:
$$S_{\mathrm{base}} = 1 - \frac{0.7\,\Delta E_{00}(F, \mathrm{skin}) + 0.3\,\Delta E_{00}(P, \mathrm{skin})}{14}$$
Lips are scored against a contrast target set by the face, where K is the lightness gap between skin and the darkest of hair and brow, and H is undertone harmony:
$$\tau = 14 + 0.42\,K$$
$$\mathrm{fit} = 1 - \frac{\big|\ |L_F - L_{\mathrm{skin}}| - \tau\ \big|}{36}$$
$$S_{\mathrm{lip}} = 0.62\,\mathrm{fit} + 0.38\,H$$
A face guide of 16 rules then adjusts the score and gives the reason shown to the shopper. For example, a pale, greyish shade on deep skin loses 0.24 with an ashiness warning, while a rich lip colour gains 0.18. A separate physical score compares finish, formula and claims with skin traits, eyelids and lips.
5. Look translation
This is our own method, and we did not find it in the literature. Pixel transfer copies a reference, and existing recommenders pick a style. Mirette stores a look as its effect on the wearer. For a face with skin s and a shade giving on-face colour F:
$$E = \left( \frac{L_F - L_s}{L_F + L_s},\ h_F,\ C_F - C_s \right)$$
The first term adapts the facial contrast measure of Porcheron et al. (2017). The others are hue, and chroma relative to the skin. The distance between two effects, with hue wrapped to the shorter way round, is:
$$d = \sqrt{(100\,\Delta\kappa)^2 + (0.4\,\Delta h)^2 + (0.5\,\Delta\chi)^2}$$
We then search the real, in-stock shades S of the same product type:
$$p^{\ast} = \arg\min_{p \in S}\ d\big(E(\mathrm{creator}, p_0),\ E(\mathrm{shopper}, p)\big)$$
If d stays above 6, the catalog has no true equivalent and the app says "closest we stock". For our look Rue Rouge, the contrast is 33% on the original face, 33% on one sample face, and 24% and 27% on two others, which are labelled closest we stock.
6. Creator match
Scores are per look, because a look only transfers if the traits that matter for it are shared:
$$M = \sum_k \alpha_k\, s_k$$
A lip look weighs skin depth 0.30, lip shape 0.30, natural lip colour 0.25 and undertone 0.15. Colours are compared with a similarity of 1 minus CIEDE2000 divided by 30. All creators in the demo are AI-made samples, and the match compares traits, not identity.
7. Product graph in Neo4j
The graph links (Trait)-[:NEEDS {weight}]->(Attribute)<-[:HAS]-(Product)-[:SOLD_BY {priceEur}]->(Seller), along with (Creator)-[:MADE]->(Look)-[:USES]->(Product), SAME_COLOUR_AS and IN_FAMILY. For shopper u and product i, the graph signal is:
$$g = \mathrm{clip}\Big(0.5 + \frac{1}{2} \sum_{t \in T_u} \sum_{a \in A_i} w_{ta}\Big)$$
Each real action moves the matching weights. A query gets 1.5 s, and after a failure the app rests the database for 30 s and uses the same logic locally, so a shopper never waits on it.
Cheaper matches need the same product kind, another brand, a CIEDE2000 of 3 or less, a saving of at least €2, and finishes that don't clash. That gives 147 products with 321 matches and a median saving of €11.95.
8. Learning from behaviour
The ranker is a logistic model with a pair weight for each trait and attribute:
$$\hat{p} = \sigma\Big(\mathbf{w}^\top \mathbf{x} + \sum_{t \in T_u} \sum_{a \in A_i} \theta_{ta}\Big)$$
Here x holds a bias, optical, physical, creator, graph and popularity. It starts from hand-set weights, so the rules work before any data exists. Each event updates it online, with learning rate 0.05 and event strengths of 1 for a seller click, 0.7 for a save, 0.6 for a try-on and 0.3 for a quick skip:
$$\delta = \eta\,\kappa\,(y - \hat{p})$$
$$\mathbf{w} \leftarrow \mathbf{w} + \delta\,\mathbf{x}$$
$$\theta_{ta} \leftarrow \theta_{ta} + \frac{1.5\,\delta}{\sqrt{1 + n_{ta}/40}}$$
Exploration is Thompson-style: noise is added to the logit, and it shrinks as evidence grows, so new products still get shown.
$$\tilde{p} = \sigma\big(\ell + \varepsilon\,s\big), \quad \varepsilon \sim \mathcal{N}(0, 1), \quad s = \frac{1.1}{\sqrt{1 + \bar{n}/15}}$$
9. Catalog pipeline
The scraper collected 997 products and 4,753 shades from 10 sellers. Each shade colour is the median of the coloured pixels in the central 60% of its image, then filtered by plausible hue. If at least 60% of a product's shades fall within a Lab distance of 5, those colours came from the packaging and are dropped. That leaves 2,159 usable colours, grouped into 28 colour families.
10. Water
The interface is a height field on the GPU, with no 3D library, stored in RGBA16F ping-pong textures. With height h, velocity v and neighbour average h-bar:
$$v' = 0.991\,\big(v + 1.6(\bar{h} - h)\big)$$
$$h' = 0.999\,(h + v')$$
Each photo sits above a waterline with a rippling reflection, and a wave carries the change between products.
Results
These results use simulated shoppers with hidden tastes, trained on 900 shoppers and 10,800 events, then tested on 200 unseen shoppers. They show that the engine can learn. They don't show that its advice is right.
Average chance of liking the top ten picks:
- Random: 23.7%
- Rules only: 38.5%
- After learning: 48.2%
By skin depth, using ITA bands (rules, then learned):
- Fair: 40.1%, then 50.2%
- Light: 38.1%, then 48.0%
- Medium: 38.6%, then 47.4%
- Tan: 37.5%, then 48.1%
- Deep: 36.5%, then 45.2%
Share of 832 foundation and concealer shades within a CIEDE2000 of 4 of each reference skin tone:
- Very light: 1.2%
- Light: 9.5%
- Intermediate: 7.7%
- Tan: 3.8%
- Brown: 2.5%
- Dark: 1.4%
Challenges we ran into
- Packaging colours: product images often show the box, so whole products came back with identical orange swatches until we added the packaging test.
- Bad labels: skincare sat in lipstick, pack sizes looked like shades, and powder blushes were filed as powder. We rebuilt the classifier.
- A wrong score on deep skin: a pale peach lip scored 98% on a deep sample face. We only caught it by looking at the render, then fixed the guide and the coverage for light shades on darker bases.
- Overclaiming: the app first said "same effect" for every translation. Measuring it showed that wasn't always true, so we added the threshold and the "closest we stock" label.
- Masks: foundation left a hard band on the forehead until the mask was feathered.
- Limited units: 54 units a scan means rationing, so sample faces, the on-device preview and the budget guard all exist to keep the demo alive.
What we learned
Learning lifted every skin depth, but it didn't close the gap: deep skin stayed about five points behind fair skin. A model can't recommend a shade the catalog doesn't stock, so fairness has to be reported by skin tone, not hidden in an average.
What's next
- Test look translation with real people, who judge whether the rebuilt look matches the original.
- Replace simulated shoppers with real behaviour.
- Replace colours estimated from images with measured swatches.
- Invite real creators
Log in or sign up for Devpost to join the conversation.