GiftFit

Try it on for someone else. The surprise stays a surprise.

Comment

Inspiration

Virtual try-on is built for people shopping for themselves. Gifts are the opposite case: the giver can't try the piece on, and asking the recipient spoils the surprise. Retailers expect 17% of holiday sales to come back (NRF, 2025), and in a 2021 survey of 3,000 Americans, 66% of jewelry-gift recipients said they want at least some input (1stDibs). We don't claim try-on fixes returns. We wanted to see whether the recipient can give that input without seeing the price, without the giver seeing their face, and without learning which piece was picked.

What it does

GiftFit is a gift assistant that a store adds with one script tag.

  • Giver, on the store. Mark-kun (a Live2D character) sits in the corner. The giver chats with the Gift Concierge (Claude), which suggests pieces from the store's catalog only, within budget, and drafts a gift brief. Default mode "Help me choose" picks 6–8 varied pieces; "I have ideas" lets the giver shortlist 1–4 designs, which are mixed with matched decoys. The giver sends a private link (copy, QR, share intents).
  • Recipient, on their phone. After consent, one guided selfie (plus an optional back-of-hand photo for rings, bracelets and watches). YouCam renders six pieces on them; they switch metals (no prices shown), heart favourites and can say "love the style, wish it came in another metal". "Delete my photo now" works at any moment.
  • Giver, the Fit Report. Ranked pieces with fit scores, short reasons, the hearted metal and price, and a buy button. Photos seen by the giver: 0. The giver never sees photos, try-on images, skin color values or face measurements, and in "I have ideas" mode doesn't learn which other pieces were shown.
  • Merchants. Paste a store URL and GiftFit builds a private, temporary preview of that store: a sanitized snapshot of its own page (no scripts, forms or tracking), marked "not affiliated", with the assistant on it, so a merchant can try GiftFit on their own catalog. On the public deployment try-on is off in previews (chat and gift flow only).
  • Any language. Mark-kun answers in the language the shopper writes or speaks. The widget, recipient page, Fit Report, e-card and emails come in 10 languages (English, Vietnamese, Japanese, Korean, Chinese, Thai, Indonesian, Spanish, French, German); the recipient page follows their browser language when it can.
  • Voice (optional). Push-to-talk with automatic language detection; spoken replies (Google Cloud Text-to-Speech) are labelled "AI voice".

How we built it

  • YouCam APIs (Perfect Corp.): AI Facial Color Tones (skin-tone-analysis) for the skin color, AI Face Attributes (face-attr-analysis, 5 features) for face shape, and Jewelry 2D VTO for earrings, necklaces, rings, bracelets and watches. One full Fit Request costs about 36 units (20 + 10 + 6 × 1). Results are cached by SHA-256 so the same input is never charged twice, and YouCam tasks are deleted through the API when the recipient deletes or the link expires.
  • Fit Engine: hearts 60, metal ↔ undertone 10 (scaled down when confidence is low), style ↔ face shape 15, budget and occasion 15. Without a try-on (previews), hearts 80 and budget/occasion 20, and no face-based reasons at all.
  • Decoys ("I have ideas"): same category, similar price, the same number of metal swatches, matched gender presentation. A simulated attacker guesses the giver's pick with probability about 0.171, close to random (1/6).
  • Claude (Anthropic): the Gift Concierge (tool use over the catalog: search, budget and metal checks, draft brief; only tool-returned products may be suggested), Fit Report reasons and the taste summary, with prompt caching, a per-IP rate limit and a daily spend cap that falls back to templates. Claude never receives photos, hex values or face measurements.
  • App: Next.js 15 (App Router), React 19, TypeScript, Tailwind CSS v4, zod. The widget runs in an iframe on GiftFit's origin (postMessage with origin checks), so the store page can't read the chat or the gift link.
  • Google Cloud: Cloud Run (two services: app and the fictional demo store), Firestore with TTL policies, a private Cloud Storage bucket (lifecycle delete after 1 day), Secret Manager, Cloud Scheduler (hourly purge), Cloud Build, Speech-to-Text V2 (chirp_3) and Text-to-Speech (Chirp 3 HD).
  • Character and voice: Live2D Cubism Web Framework + unmodified Cubism Core with the "Mark-kun" sample model (an SVG mascot is the fallback). Voice is Google Cloud only: Speech-to-Text V2 (chirp_3, automatic language detection) and Text-to-Speech (Chirp 3 HD where the language has it, else Neural2/Standard, else text only).
  • Store preview: reads only public product data (JSON-LD, Open Graph, sitemap, Shopify products.json, WooCommerce Store API), respects robots.txt, with SSRF guards (https only, private/metadata addresses refused, connect to the resolved IP, redirects re-checked). The page snapshot is sanitized (no scripts, event handlers, forms or iframes), served under a strict Content-Security-Policy inside a sandboxed iframe with no script permission, marked noindex, and deleted after 24 hours. Login and payment pages are refused, and pages with fewer than 3 jewelry products fall back to a neutral template.
  • Tests: Vitest (2,326 tests at the last gate) and Playwright end-to-end suites (API, UI, cross-origin plugin), including an independent scanner that fails if any giver response carries an image link, hex color or face field.
  • Imagery: the Aurum catalog (161 designs, 179 metal variants) was drawn in code (SVG/PNG); 12 product photos and 18 metal-variant edits were AI-generated with Google Flow.

Challenges we ran into

  • Undertone from one face color is weak. On five AI-generated portraits spanning Fitzpatrick I–VI as prompted, the real API's skin color hue spanned only 57.9–64.3°. Only 1 of 5 estimates matched the prompt's intended undertone, and one was a moderately confident miss (0.54). We changed the design instead of hiding it (see "Fairness and limits" below).
  • Try-on capture rules. Earrings failed when hair covered the ears (engine error "earlobe alignment not confident"); bracelets and watches failed when the wrist wasn't centred. The capture guide now asks for tucked hair and a centred wrist with some forearm.
  • Surprise leaks. Our first men's demo carousel had decoys with no try-on image, and later decoys that read as feminine next to a watch and chains. Both let you spot the giver's picks at a glance. Decoys now match rendering and gender presentation, and a leak check finds no attribute separating picks from decoys.
  • Live2D on real GPUs. Mark-kun failed on an Intel GPU because a load timeout disposed the model while shaders were still compiling. We prefetch and validate shaders and show the SVG mascot until the first Live2D frame.
  • A natural voice vs. a fast one. We first ran an open-source voice model in the shopper's browser (about 399 MB to download), then tried it on our server, where its response time varied up to 2.7× between fresh instances. We dropped it for Google Cloud Text-to-Speech: no download, a smaller server image, and voices in many more languages (text only where none exists).
  • Live2D start-up. Mark-kun took 7.4 s to appear on a cold load because the framework compiled 237 shader programs we never used. Compiling only what the model needs brought the median to about 2.4 s.
  • A real photo, refused. On a real public store, YouCam's analysis refused a full-frame selfie as "face tilted too far downward". Every earlier pass had used a face crop. The analyses now read a face crop (the try-ons keep the full photo), a refused half retries alone on the full frame, and the recipient gets a specific hint ("hold the phone at eye level, chin slightly up") in their language. YouCam refunds failed tasks, so the refused attempt cost nothing.
  • Bundled vs unbundled. Our first Firestore deploy failed in the bundled server only; traffic went back to the previous revision within minutes. Storage changes are now verified on the bundled server and deployed as a no-traffic candidate first.

Accomplishments that we're proud of

  • The giver gets enough to buy with confidence and nothing about the recipient's face: an automated test fails if any giver endpoint returns an image link, hex color, CIELAB value or face field.
  • "Delete my photo now" removes the selfie, the hand photo, every try-on image and YouCam's task copies; tests check that no recipient image URL resolves afterwards.
  • A recipient's decoy guess is about as good as chance (≈ 0.171 vs 1/6).
  • One script tag on a separate-origin store brings the character, chat, voice and the whole gift flow.
  • A judge demo that runs the full recipient and giver journey without a selfie and without spending YouCam units.
  • The whole flow on a real public jewelry store we have no relationship with: GiftFit read its public catalog, Mark-kun suggested real pieces, and YouCam rendered six of them on an AI-generated model, for 36 units (shown in the video, store name masked).

What we learned

  • A single skin color from a face carries little undertone signal. Asking the recipient, and letting their hearts decide, is fairer than guessing.
  • Privacy has to be enforced at the API boundary, not just in the UI.
  • Surprise is a security property: anything that differs between picks and decoys (rendering, style, swatch count) gives the pick away.
  • Try-on quality depends as much on the capture guide as on the API.

What's next

  • Try-on on preview stores for supported jewelry categories (it's off on the public deployment because each try-on is a paid call; the video shows it working in an owner-only session).
  • An on-device head-pose check before upload, so a tilted selfie is caught before any API call.
  • Admin beyond v0: accounts, localized admin pages, occasion tags for merchant products (today every merchant item reads as "not tagged" for the occasion).
  • A native-speaker review of the nine non-English languages (they are machine-translated today).
  • More fairness samples before any accuracy claim about undertone.
  • A trademark search on the name before any use beyond the hackathon.

Fairness and limits

  • Undertone is an estimate the recipient can correct. YouCam returns color values, not an undertone. GiftFit estimates warm, neutral or cool from the CIELAB hue angle relative to skin depth; the thresholds are our own (no published cutoffs exist).
  • What we measured. Real YouCam output on five AI-generated portraits (Fitzpatrick I, III, IV, V, VI as prompted): hue 57.9–64.3°; 1 of 5 matched the intended undertone; 4 of 5 had confidence ≤ 0.20. Any hue in 55–67° now returns "neutral" at confidence 0.2, so there are 0 confident misses on those samples, at the cost of more "neutral" answers (45% misclassification on our synthetic swatch harness, almost all low-confidence neutral). In that harness, Fitzpatrick V–VI read cooler by 10.3%, below its flag baseline.
  • So: the recipient sees "estimate", is asked to pick warm, neutral or cool, and their answer wins; the metal factor is worth at most 10 of 100 points and shrinks at low confidence; no metal is ever marked "avoid"; hearts carry the most weight. Face shape is also labelled an estimate.
  • Try-on images are AI previews, not a sizing guarantee. Public store previews have no try-on, so their Fit Reports make no face-based claims and label untried pieces "Not tried on".
  • Five samples are not enough for an accuracy claim, and we don't make one.

Privacy in one paragraph

The recipient's photo is kept for at most 1 hour, only so they can try other metals; it's deleted when they tap Done, after 4 extra metal try-ons, or at once with "Delete my photo now", and it's never shown to the giver. Try-on images are served only to the recipient through links that last at most 15 minutes and only work in their own session. Every gift link expires after 72 hours, and "Delete everything" removes the recipient's estimates, hearts and report at any time. Email addresses typed into the widget are stored sealed (AES-256-GCM) and deleted with the request; email sending is switched off on the demo deployment.

AI-generated content and credits

  • Imagery: all products are from Aurum Jewelry, a fictional store. Product images are either drawn in code (SVG/PNG, CC0) or AI-generated with Google Flow from the team's text prompts. "Lumen & Co.", the demo store, is fictional too.
  • People: every person shown is an AI-generated model made by the team, labelled "AI-generated model · demo" in the app. No real person's photo is in the repo, the judge demo or the gallery; the real store's own lifestyle photos are blurred in the video.
  • Voice: spoken replies are machine-generated by Google Cloud Text-to-Speech and labelled "AI voice". The video's narration and the shopper's voice in it are AI-generated too.
  • Real store in the video: one scene uses a real public jewelry store to show GiftFit on a catalog it didn't build. Its name, logo and address are masked, it is not affiliated with GiftFit and didn't endorse it, and the scene was recorded in an owner-only session. Product photos remain the store's property.
  • YouCam API by Perfect Corp.: Facial Color Tones, Face Attributes and Jewelry 2D VTO. GiftFit is not made or endorsed by Perfect Corp.
  • Claude by Anthropic: Gift Concierge, Fit Report reasons and taste summary.
  • Google Cloud Speech-to-Text and Text-to-Speech for voice input and spoken replies.
  • Live2D: This content uses sample data owned and copyrighted by Live2D Inc. The sample data are utilized in accordance with terms and conditions set by Live2D Inc. This content itself is created at the author's sole discretion. Live2D Cubism Core and the Cubism Web Framework are under Live2D's own licenses and are not committed to our repo. Live2D and Cubism are trademarks of Live2D Inc.; GiftFit is not made or endorsed by Live2D Inc.
  • Code: MIT license.

Built With

  • anthropic
  • claude
  • cloud-build
  • cloud-scheduler
  • firestore
  • google-cloud
  • google-cloud-run
  • google-cloud-speech-to-text
  • google-cloud-text-to-speech
  • google-flow
  • live2d-cubism
  • next.js
  • node.js
  • perfect-corp
  • playwright
  • react
  • secret-manager
  • tailwindcss
  • typescript
  • vitest
  • youcam-api
  • youcam-face-attributes
  • youcam-facial-color-tones
  • youcam-jewelry-2d-vto
  • zod
Share this project:

Updates

Submission history