Know Your Skin

Website view
Mobile view

Inspiration

We were familiar with Korean skincare and always had friends often ask about how we had clear skin and recommended products.

What it does

A web app that classifies your specific acne type and returns a tailored morning and evening routine plus top-rated Amazon products matched to the ingredients you actually need.

How we built it

The model is a fine-tuned MobileNetV3-Large on the Roboflow acne classification dataset (~1,755 images, 6 classes). We used class-weighted cross-entropy with label smoothing, WeightedRandomSampler for imbalance, aggressive augmentation (rotation, color jitter, random erasing), and cosine-annealing LR with warm restarts to create this model and achieve a ~74% validation accuracy, which is competitive for this task.

We exported PyTorch and used ONNX, shipped in the repo, and ran entirely in-browser via onnxruntime-web using its WASM backend with SIMD. We also used softmax on logits, filtered out cysts at the source, and created a majority-vote across each pass system.

To track the face, we used MediaPipe FaceLandmarker for a 478-point face mesh and the facial transformation matrix. We decomposed the matrix into yaw/pitch/roll Euler angles in real time, smoothed them with a 0.25 alpha lerp, and gated the scan on pose tolerance.

For the live scan, we created a seven-stage system with pose targets. The sweep line on the mesh only advances while the user's pose matches, so each of the 20 captures is genuinely from a different confirmed angle. On completion, we run majority voting by counting top-1 class votes across passes, to boost the winner's confidence by its vote share so a strong consensus survives the clear-skin threshold.

We provide live guidance (turn head, lift chin, find better light) until the algorithm can confirm alignment and lighting for 1.4 seconds, then auto-capture. The user can also draw on the still photos for certain sections to be aware of, and we run 8 augmented variants (horizontal flip, small rotation/scale/translate) through the model and majority vote, making it a better outcome than just one single pass through.

To check the lighting, it draws the video frame to a 32×24 canvas to compute the luminance (0.299R + 0.587G + 0.114B), running average over 8 samples. If it is under the threshold, it pauses the scan and prompts a "find brighter lighting" text.

Challenges we ran into

We struggled with finding a good dataset to train the model. We also had hardware limitations, as our laptops weren't very compatible and were very slow at teaching the model