Skin Journey

Snap a selfie and get clinical-grade acne analysis! Our AI tracks your skin over time and tells you whether to adjust your routine or see a dermatologist, all from your own phone camera.

Comment

Inspiration

We've all stared in the mirror, wondering if that new skincare product is actually working or making things worse. With countless products promising miracles and dermatologist appointments often weeks away, people are left guessing about their skin health.

The inspiration for Skin Journey came from our personal struggle of tracking acne progress over months of treatment, but having no objective way to measure improvement. Photos were scattered across our camera rolls, but were impossible to compare. Meanwhile, severity grading systems used by dermatologists weren’t fully accessible.

We realized we could bridge this gap with computer vision and machine learning, building a system that tracks skin changes over time and provides the kind of objective assessment that typically requires a month-long wait.

What it does

Skin Journey AI is a mobile-first acne tracking system that turns your phone into a dermatology analysis tool. Here's what happens when you take a selfie:

  1. Lesion Detection: A U-Net segmentation model identifies acne lesions on your face, creating a detailed probability map of inflammation areas
  2. Severity Grading: The system classifies your acne as mild, moderate, or severe (0-10 scale), using both local ML models and OpenAI Vision confirmation
  3. Eczema Detection: Analyzes patterns that may indicate atopic dermatitis or eczema-like inflammation
  4. Smart Recommendations: Provides rule-based guidance on whether you should try an over-the-counter product change or escalate to a dermatologist
  5. Timeline Tracking: Builds a visual history of your skin journey, making progress (or regression) clear

How we built it

Architecture

We split the system into three components:

  • Mobile Client (React Native/Expo): Handles image capture, displays analysis results, and maintains the timeline UI
  • FastAPI Backend: Orchestrates the ML pipeline and serves predictions
  • ML Workspace: Training ground for our segmentation and severity models

The ML Pipeline

We built a multi-stage pipeline:

Stage 1: Lesion Segmentation

  • Started with the Acne04-v2 dataset, which provides expert annotations as {cx, cy, radius} circles
  • Initial approach: Rasterized filled circles as training masks—simple but crude
  • Breakthrough: Integrated Segment Anything Model (SAM) to convert expert circles into pixel-perfect lesion boundaries. We used the circle annotations as prompts to SAM, generating masks that actually follow lesion contours instead of filling geometric shapes python # SAM-refined masks dramatically improved segmentation quality python -m ml.datasets.acne04v2_sam_masks \ --sam-variant mobile_sam \ --size 512 --max-side 1024
  • Trained a lightweight U-Net on these SAM-refined masks, achieving decent Dice/IoU metrics while staying mobile-friendly

Stage 2: Severity Classification

  • Extracted 12 features from the U-Net predictions:
    • Probability mass and thresholded area
    • Connected component counts (lesion density)
    • Redness statistics in HSV space
    • Texture/edge features
    • Interaction terms between redness and lesion intensity
  • Built an MLP "severity head" trained as a 3-class classifier (mild/moderate/severe)
  • Used balanced batching to prevent collapse toward majority classes, which was crucial when training on imbalanced medical data

Stage 3: OpenAI Vision Integration

  • Added GPT-4 Vision as a confirmation layer for acne severity and eczema patterns
  • Dual-model approach allows for fast local inference with cloud-based validation

Challenges we ran into

1. The Mask Quality Problem

Our first U-Net trained on simple rasterized circles produced okay results, but the boundaries were too soft and the segmentation wasn’t precise. We realized the training targets themselves were the bottleneck—filled circles are terrible representations of actual lesion shapes.

Solution: We pivoted to SAM-based mask generation. By treating expert annotations as prompts rather than ground truth shapes, we generated training data that actually captured lesion morphology.

2. Severity Head Collapse

Initial severity classifiers would just predict "mild" for everything because that was the majority class in our dataset.

Solution: Implemented balanced batch sampling that oversamples rare severity levels during training. The --balanced-batches flag became really important for this task.

3. Real-World Lighting Variations

Lab datasets are clean, while real selfies rarely are. We saw the model struggle with harsh shadows, different skin tones, and varying phone camera qualities.

Solution: Extensive data augmentation during training (brightness, contrast, hue shifts) and preprocessing in the API pipeline. The OpenAI Vision fallback also helps catch edge cases the local model misses.

4. Mobile Performance

Running deep learning models on phones is hard, and we needed predictions that were fast enough for a smooth user experience.

Solution: We kept the U-Net architecture lightweight, used MobileSAM for mask generation instead of full SAM, and optimized the API response payload to minimize data transfer.

Accomplishments that we're proud of!

  • Built a production-grade ML pipeline that goes from raw dataset to deployed models with clear evaluation metrics
  • Integrated research (SAM) into a practical workflow that improves training data quality
  • Created a genuinely useful health tool that could help people make better decisions about their skincare
  • Achieved decent segmentation accuracy on a challenging medical imaging task with limited training data
  • Designed for reproducibility: The entire ML workflow is scripted so anyone can download the dataset, train the models, and reproduce our results

What we learned

Technical Lessons

  • Using SAM to improve our training data was a huge help. We learned that sometimes the best use of a large model isn't inference, but creating better training sets for smaller models
  • Even with deep learning, the features in our severity head (like redness × lesion intensity interactions) captured domain knowledge the U-Net alone couldn't learn
  • Class imbalance is brutal in medical ML, especially in a lot of open source datasets

Product Lessons

  • The local + cloud architecture means the app works even when OpenAI is down
  • We learned to favor false positives in dermatologist recommendations
  • Timeline visualization is really important, since users don't just want a score; they want to see progress over weeks

What's next for Skin Journey

  1. Expand detection capabilities and improve our models: Add support for rosacea, hyperpigmentation, and other common conditions and more evenly train our models for better outcomes
  2. Product tracking: Let users log which products they're using and correlate with skin changes
  3. Environmental factors: Track weather, diet, stress levels to identify triggers
  4. Retinol response monitoring: Special tracking for users starting prescription retinoids (where progress takes months)
  5. Use LSTM/Transformer architectures to predict future skin state based on historical trends

Built With

Share this project:

Updates

Submission history