Inspiration

We build online stores for small businesses, and beauty shops keep telling us the same thing: shoppers stand in front of forty serums, ask "which one is for me?", and leave without buying when nobody is there to answer. Skin analysis apps exist, but they recommend from their own brand catalog, not from the shelf the shopper is standing in front of.

What it does

The shopper takes a selfie (or a judge taps "Use sample selfie"). Shelf Check sends it once to the YouCam File API and runs two YouCam tasks on it: the Skin Analysis API measures eight concerns (breakouts, pores, redness, oiliness, dehydration, fine lines, dark circles, dark spots) and the Fitzpatrick Skin Type API classifies the skin from type I to VI. The app shows the skin map (tap a concern to see the API's own mask over your photo), the overall score, the estimated skin age and the skin type, with one line on what that type means: on types I-II sun damage shows first, so fine lines, redness and spots weigh more; on types IV-VI the marks left by breakouts (post-inflammatory hyperpigmentation) and visible dryness lead, so those weigh more. Then it picks the three products, from this store's own catalog, that cover the most of the shopper's weighted need. Every pick says why ("For: Dehydration, Redness"), so the shopper trusts it and the store sells what it actually has on the shelf.

Any Shopify store's real shelf. Under "Whose shelf?" a store owner types the store's address (try https://shelf-check.designmaster.workers.dev/?store=sokoglam.com) and the picks come from that store's live catalog: in stock, priced, linked to the product page, and justified by the store's own words ("Store says: niacinamide, barrier"). The same scan gives the owner a shelf map, how many products claim each of the eight concerns the YouCam API measures, with the gaps highlighted. On sokoglam.com, 131 of 250 listed products made the shelf, 86 claim hydration and only 5 claim dark circles. When a shopper's low-scoring concern has nothing on the shelf, the page says so instead of pushing something unrelated.

How we built it

The YouCam flow is File API (pre-signed upload) -> skin-analysis task (SD, 8 concerns, mask overlay, 12 units) and fitzpatrick-scale-analyzer task on the same file, run in parallel -> poll both until success. For each concern, need = (100 - raw_score) x skin-type weight; weights never go below 1, so a measured need is never hidden, the type only says which needs matter more for that skin (e.g. dark spots x1.3 on type V, fine lines and redness x1.2 on type I). Each catalog item scores the sum of the needs it treats, ties go to the cheaper item. The skin type is advisory: if its task fails (face angle, lighting) the shopper still gets the skin map weighed as measured.

The shelf is either the demo catalog.json or a Shopify store's public /products.json (first 250 products) and /meta.json (name, currency). A product makes the shelf only if its own title, product type or tags claim one of the eight measured concerns: concern tags like Skin_Concern::Dryness/Hydration, actives like niacinamide, salicylic acid, retinal or tranexamic acid, words like "eye cream" or "SPF". Lip, body and hair products, sold-out items and same-name duplicates are dropped. The shelf loads before any YouCam unit is spent, only public store addresses are fetched, and every store-supplied string is escaped before it reaches the page. The live app runs on Cloudflare Workers (worker.js) with a dependency-free Python twin (server.py) and a parity test that checks the Worker ranks exactly like the Python reference for all six skin types; the Shopify import is tested against a real captured catalog.

Challenges we ran into

The engine rejects faces that are too small (error_src_face_too_small): the face has to fill more than 60% of the frame, so the sample photo had to be cropped. The mask URLs returned by the task are signed and expire in two hours, so the demo serves the captured masks from the app itself instead of dead links. Real catalogs are messy: on sokoglam.com 13 of the first 30 listings were sold out and one was a $0 bundle builder, on kravebeauty.com a lip treatment ranked first for dehydration, and on versedskin.com the same product showed up twice under two listings, so stock, price, face-only and duplicate filters run before ranking. Shopify also answers HTTP 429 to bursts of requests, so each store's catalog is read at most once an hour and a busy store shows its last good read, labeled with the time it was read.

Accomplishments that we're proud of

A live public app that answers in about seven seconds with a real YouCam call, a recommendation that is explainable line by line (every pick shows which measured concerns justified it and which words of the store's own listing back it), and a shelf any Shopify beauty store can plug in without sending us a file.

What we learned

The raw scores are precise enough to rank products directly, and turning "score" into "need" makes the recommendation feel like advice instead of a diagnosis. Reading a real catalog showed the other side: many stores do not say in their listings what a product is for, and the shelf map makes that visible to the owner.

What's next for Shelf Check

The same import for Nuvemshop and WooCommerce stores, let the store tune the skin-type weights for its own range, add the Apparel VTO step for the clothing side of the same store, and an in-store kiosk mode for a tablet next to the shelf.

Built With

Share this project:

Updates

Submission history