Capy Shelf

Is this right for your skin? Scan once with YouCam AI Skin Analysis, snap any ingredient label, and Capy gives a calm, explained verdict right in the aisle.

Comment

Inspiration

People wonder about their skin right before a purchase: standing in the skincare aisle with a product in hand, unable to read a 40-item ingredient list, remembering the cream that once broke them out or dried them out. Skin analysis tells you about your skin, and labels tell you about the product, but nothing connects the two at the moment you decide. Capy Shelf does.

What it does

Capy Shelf is a calm shopping buddy. Capy, a capybara riding in the shopping basket, answers one question: is this product right for my skin? No shaming, no "flaws", plain words.

  1. Scan once. A selfie, guided by the YouCam JS Camera Kit (live checks for lighting, pose and face position, then automatic capture) or uploaded, goes to the YouCam AI Skin Analysis API. Capy keeps the concerns that decide purchases: acne, pore, oiliness, moisture, redness, texture and skin type (whole face, T-zone and U-zone), each with a score and YouCam's detection mask over the photo.
  2. Snap the label. A vision language model reads the ingredient list off the photo, in order. Every ingredient becomes a chip colored for this person's skin.
  3. Get a verdict. A deterministic ingredient table with cited sources meets this person's measured scores and returns Good match, Use with care or Skip for you, the ingredients that drove it with their position on the list, and where on the face the related concern was seen, computed from the YouCam mask pixels. The language model only words the explanation; it never decides.
  4. Compare. Two or three products side by side, then a tiny cleanse, treat, moisturize routine from the ones that fit, with "what to look for instead" where nothing does.

The aha: the same moisturizer is a Good match for Shopper A, whose main focus is hydration, and Skip for you for Shopper B, whose scan shows pores and breakouts mostly on the cheeks, because isopropyl myristate is third on its list and may clog pores. Same label, same rules; only the measured skin differs.

How we built it

  • YouCam AI Skin Analysis, server-side. File API upload (POST /s2s/v2.0/file, then a PUT to the signed URL), one skin-analysis task with seven SD dst_actions (acne, pore, oiliness, moisture, redness, texture, skin_type, 12 units), then polling until success. Documented photo errors such as error_src_face_too_small become plain advice, and a failed task gives the live scan back because YouCam only charges on success.
  • YouCam JS Camera Kit in skincare mode for live selfies, so photos already meet the analysis rules and fewer scans fail.
  • Bands from raw scores. Capy decides on raw_score (70 and up looks balanced, 50 to 69 worth watching, under 50 main focus) and shows YouCam's friendlier ui_score too. Skin type can raise a matching concern by one band, never lower it.
  • Where on the face, from mask pixels. The moisture mask covers the whole face, so its bounding box becomes the face box, and every other mask's alpha is split into forehead, nose, cheeks and chin.
  • A verdict you can audit. Each ingredient is matched to effect tags (harsh foaming agent, drying alcohol, fragrance, may clog pores, water magnet, barrier support, BHA, niacinamide and more). A risk only counts for a concern that is not balanced for this person, weighted by strength, band and position on the list. Every tag has 1 to 3 sources (AAD, FDA and eCFR monographs, EU cosmetics regulation, PubMed), shown on an in-app How it works page. Unknown ingredients are listed as "not judged", never guessed.
  • Qwen 3.8 on W&B Inference (OpenAI-compatible) reads labels and writes the explanation from a list of facts, with rules against new claims.
  • Replay of a recorded run. Every YouCam and LLM result in the demo was recorded once from the real APIs and ships in the repo, so judges can click through without keys. Live mode runs your own selfie and labels.
  • Next.js 16, React 19, TypeScript, Tailwind CSS 4. Built to WCAG 2.2 AA: axe-core reports 0 violations on every page at 1440 px and 390 px.

Challenges we ran into

  • Where on the face, without landmarks. The API returns masks, not landmarks. We used the full-face moisture map as the face box and split the other masks into zones. The first split called jawline breakouts "chin"; narrowing the chin zone to the lower center made the result (89 percent cheeks) match the photo.
  • Keeping the AI honest. The explanation model once credited a serum for pores because of its product name. We moved every decision into deterministic code and gave the model only facts.
  • A shared unit budget. 1,000 units across two projects: each demo scan was recorded exactly once, live scans are capped per server, and the whole UI was built against fixtures. Total spent: 48 units.

Accomplishments that we're proud of

Every verdict names the ingredient, its position on the label, the measured score behind it and the area of the face, so a shopper can see exactly why. And it stays kind: Capy explains that oily skin can still be low on water instead of calling anything a flaw.

What we learned

Skin can be oily and dehydrated at the same time, and the API measured exactly that on our first scan. Raw scores and friendly scores answer different questions, so we decide on one and show both.

What's next for Capy Shelf

HD analysis for regional pore scores, barcode lookup to skip the label photo, and shelf ranking: scan once, then see the three best matches in an aisle with reasons. For retailers, a shelf-edge or in-app assistant that recommends with reasons instead of hype.

Honest limits

Not medical advice and not a diagnosis. Ingredient effects are tendencies, not guarantees. Capy suggests a patch test for anything new and a dermatologist for breakouts that are painful, scarring or do not settle. All products and brands in the demo are fictional; demo faces are licensed stock photos.

Built With

  • assemblyai
  • axe-core
  • ffmpeg
  • next.js
  • openai-sdk
  • playwright
  • pngjs
  • pnpm
  • qwen
  • react
  • tailwindcss
  • typescript
  • wandb
  • youcam-ai-skin-analysis
  • youcam-api
  • youcam-camera-kit
Share this project:

Updates

Submission history