Baseline

A skin measurement instrument, not a beauty app. It answers one question — did a skincare product or habit actually change this person's skin? — and it's built to say "I can't tell" out loud whenever that's the honest answer, instead of manufacturing a verdict.

Inspiration

Every skincare-tracking product I've seen shows a score and lets the user infer meaning from it. That's backwards for anything claiming to measure change: a single photo's score is a mix of real skin condition and that day's lighting, angle, and camera noise, and nothing about a bare number tells you which part is which. I wanted to build the opposite of a beauty filter — something that starts from the assumption that most of what a skin-analysis API reports on any given day is noise, and only says something changed when the evidence clears that noise, measured on the person's own repeat photos rather than assumed.

Perfect Corp's YouCam Skin Analysis API supplies the sensor. SerpApi supplies the thing being tested against: not "did some number go up," but "did the manufacturer's own specific, time-bound claim about this product hold up for this person," retrieved live rather than typed in by hand or invented.

What it does

1. Baseline session: the user takes 8 photos in one sitting. Perfect Corp's Skin Analysis API scores each on every measurable concern. The spread across those 8 repeat readings of an unchanged face becomes that concern's noise floor — the smallest delta the instrument will ever call real for this person.
2. Feasibility report, before anything is tested: npm run feasibility reports, per concern, whether a realistic future change could ever be told apart from that noise floor — and refuses to pretend otherwise when it can't.
3. Experiment pre-registration: name a product or habit, and SerpApi retrieves the manufacturer's actual claim, with source URL and timestamp. The claim maps to one measurable concern (shown with its reasoning, always overridable), and the minimum detectable change is stated and frozen before any follow-up photo exists.
4. Follow-up comparison: a later session's mean is compared to baseline. A verdict — moved_up / moved_down / no_change / indistinguishable — is reported per concern, annotated (never silently overridden) when the follow-up sitting itself looks statistically noisier than the baseline.

Design rules that hold throughout: only raw_score ever feeds a computation, never ui_score. No comparison to population norms — only a person against their own baseline. No thresholds, categorical labels, or medical terms — direction is reported as numeric sign only, never "better" or "worse," since several concerns invert which direction is desirable and this instrument has no basis to make that call. A change smaller than the noise floor is "indistinguishable," explicitly not "no change" — those are different claims and only one of them is backed by evidence.

How we built it

• Perfect Corp YouCam Skin Analysis (S2S API) as the sensor. A real client (src/perfectcorp/client.ts) built and corrected against the live API, not just the docs — see the Perfect Corp submission for the four documented-vs-live discrepancies this surfaced.
• A statistics engine built from Node builtins only (src/analysis/): noise-floor estimation (max(2·sd, range)), an F-distribution implementation (regularized incomplete beta via Lentz's algorithm, log-Gamma via the Lanczos approximation) backing a proper two-sample variance test, and a feasibility module that turns "headroom to the top of the scale" versus "noise floor" into an honest measurable / marginal / noise_too_large / no_headroom classification.
• SerpApi for claim retrieval and footnote-context lookup, both behind a disk cache shared with the Perfect Corp client, with a brand-token guard against misattributing an unrelated product's claim.
• A Next.js (App Router) web app for guided capture (Perfect Corp Camera Kit with a getUserMedia + MediaPipe face-landmark fallback), session review with per-concern mask overlays, and the experiment declare/freeze/progress flow.
• 94 unit tests (Node's built-in test runner via tsx) covering the statistics engine, the claim/footnote extraction logic, and the full experiment declare → map → detectability → freeze pipeline — all passing.

The strongest result: the instrument disqualifying its own metrics

Running the real feasibility check against a real 8-capture baseline session (data/sessions/session01.json): acne's noise floor (15.59) exceeds the entire 11.85 points of headroom left to a perfect score — no realistic improvement, however successful, could ever be told apart from ordinary shot-to-shot noise for this concern, on this face. redness, age_spot, and wrinkle come back no_headroom — this person's baseline already sits so close to the top of the scale that there's essentially nowhere left for a number to move. The instrument reports this before any follow-up session is ever taken, rather than silently producing a comparison later that looks meaningful but isn't. I consider this the project's actual thesis, demonstrated on itself: a measurement tool that will name its own blind spots is more trustworthy than one that always has an answer.

Replacing an ad-hoc threshold with a real statistical test — a trade, not a fix

The original design for flagging "was this follow-up sitting itself unreliable?" compared the two sessions' max(2·sd, range) spreads against a fixed 1.5× multiplier. That comparison uses two numbers built for a different job — a conservative reporting floor, not a variance estimator — against an arbitrary constant.

This is now reproducible by anyone who clones the repo, not just asserted: scripts/validate-reliability-test.ts (npm run validate:reliability, documented in docs/reliability-test-validation.md) runs a 20,000-trial-per-n Monte Carlo directly against the real, shipped fUpperTailPValue and the real LOW_CONFIDENCE_ALPHA constant — it cannot silently drift from what actually ships, because it imports them rather than reimplementing them.

False positives (baseline and follow-up drawn from the identical distribution — any flag is spurious):

n	old rule (>1.5×)	F-test (p<0.05)
3	30.98%	5.04%
4	26.55%	5.05%
5	23.00%	5.12%
8	16.79%	4.96%

Statistical power (follow-up genuinely noisier — the flag should fire):

n	true sd ratio	old rule	F-test
3	2×	63.17%	17.32%
8	2×	75.27%	52.77%
3	3×	80.33%	32.41%
8	3×	94.81%	86.77%

This was not a bug fix — it was a deliberate trade of recall for precision, and it should be described as one. The old 1.5× rule actually caught a genuinely bad sitting far more often than the F-test does: 63% versus 17% at n=3 for a follow-up with twice the baseline's true noise. What the old rule bought that detection power with was crying wolf on perfectly good sessions 23–31% of the time at n=3–5 — roughly one in three "your follow-up sitting was noisy" flags were simply wrong. A flag that fires on a third of good sessions trains the user to ignore it, so the F-test was chosen specifically to fix precision at a real, known cost to recall: its false-positive rate sits within half a point of its 5% target at every tested n, and in exchange it misses most bad sittings at small n — catching only 17% of a 2×-noise follow-up and 32% of a 3×-noise follow-up at n=3.

The real conclusion is not that the F-test solved the reliability problem — it's that neither rule is well calibrated at these sample sizes, because 8 captures is too few to estimate a variance ratio precisely. A test that misses four out of five bad sittings does not protect much either; it has simply stopped lying about good ones. The genuine fix is more captures per session, not a better threshold — sensitivity is still climbing at n=8 (52.77% and 86.77% in the table above) with no sign of having reached a usable level. Saying that plainly is stronger than claiming the reliability problem is solved, and it's the same thesis this whole project applies everywhere else — an instrument that reports the limits of what it can tell — turned on its own internals.

Challenges

• The vendor's own display score actively works against the product's purpose. ui_score = 41.19 + 0.5069 × raw_score (R² = 0.915, n=96 real paired readings) — using it for anything would mechanically shrink every measured delta by roughly half, nonlinearly and inconsistently across concerns, in a way that can't be corrected after the fact. This had to be avoided at the source (raw-score-only), not patched downstream.
• Perfect Corp's JS Camera Kit required reverse-engineering to integrate. The public docs described an API surface (window.YMKAsyncInit) that doesn't exist in the shipped bundle; getting a working capture flow required decompiling the actual sdk.js to find an undocumented required mount element (<div id="YMK-module">) and a hardcoded desktop fallback display size (360×480) that silently applies whenever width/height are omitted from YMK.init(). Full round-by-round investigation, including a real regression traced to a CSS change and reverted: docs/capture-design.md.
• A fictitious product query returned a real, unrelated product sharing only a generic marketing word with the query — the kind of false attribution a naive claim extractor would produce silently. Closed by requiring a brand-token match, not just keyword overlap.
• Safari doesn't support the Camera Kit capture pipeline (A MediaStreamTrack ended due to a capture failure) — detected up front and surfaced as an explicit banner rather than a silent, confusing fallback.

Accomplishments

• A working, tested pre-registration flow that freezes a primary outcome and a minimum-detectable-change threshold before any follow-up data exists — not after the fact, when it would be too easy to rationalize whatever moved.
• 94 passing unit tests across the statistics engine, the claim-extraction logic, and the full experiment lifecycle.
• Two independently interesting, numerically-backed findings about a real third-party API (the ui_score compression regression and the four docs-vs-live discrepancies) that exist because this project chose to distrust its own primary data source and check.

What we learned

That the hardest part of building an honest measurement tool isn't the statistics — it's resisting the pull to make every output look more decisive than the underlying data supports, and resisting the pull to describe a trade-off as a fix. Replacing the 1.5× rule with the F-test wasn't "solving" the reliability check — it was choosing not to cry wolf on good sessions, at the explicit cost of missing most bad ones at the sample sizes this instrument actually uses. That trade, the confounded camera-comparison, and the vendor's own range-compressed display score were all found by explicitly asking "is this as strong as it looks, and what did it cost," and each answer became a documented limitation rather than a rounded-up claim.

What's next

• Run an actual follow-up session against the frozen Hydro Boost experiment once the 10-day window elapses, to see what the instrument reports on real longitudinal data rather than a single baseline sitting.
• Design a controlled version of the guided-vs-handheld capture comparison (same camera, guidance on/off) to de-confound the ~38% median noise-floor improvement observed so far.
• Collect noise floors and feasibility thresholds from more than one person to find out whether src/analysis/feasibility.ts's thresholds (explicitly flagged in-code as "a starting heuristic derived from one user's data") generalize at all.
• Resolve the one remaining open item in the Camera Kit integration: confirming a live end-to-end capture in an environment with a real camera Camera Kit's anti-spoofing filter will accept (this development sandbox has none).
• Increase captures per session and re-run npm run validate:reliability against the higher n — the Monte Carlo validation shows detection power still rising at n=8 with no sign of plateauing, so more captures is the actual lever for a well-calibrated reliability check, not a further-tuned threshold.

Honest limitations, stated plainly

• Single user, single instrument, single room. Every noise floor, every feasibility threshold, and the guided-vs-handheld comparison are measured on one person's face, one camera setup at a time. None of it is a universal constant.
• The guided-capture noise reduction is confounded. Camera and capture method changed together; the improvement cannot be attributed to guidance alone.
• Neither reliability rule is well calibrated at n=3–8 captures per session, because 8 captures is too few to estimate a variance ratio precisely. The F-test fixed the old rule's 23–31% false-positive rate down to ~5% at the cost of missing most genuinely bad sittings (17.32% detection at n=3 for a 2×-noise follow-up, rising to only 52.77% at n=8) — a deliberate trade of recall for precision, not a solved problem. The actual fix is more captures per session, not a better threshold.
• No product has actually been proven to work. The Hydro Boost experiment is declared and frozen, not concluded — there is no follow-up data yet, and the entire point of freezing the primary outcome first is so that whatever the eventual answer is, it wasn't chosen after seeing the result.

Built With

• ffmpeg
• hyperframes
• mediapipe
• next.js
• node.js
• perfect-corp
• react
• serpapi
• tailwind
• typescript
• youcam-api