AuraMirror Android: Skin AI & eCommerce Companion

Inspiration

Most mobile skincare analyzers only grade a static photo without guiding users to the right active ingredients or localized shopping options for their real-time skin condition. We built AuraMirror Android natively in Kotlin and Jetpack Compose to combine Perfect Corp.'s YouCam Skin AI API with hardware ambient light sensing (Sensor.TYPE_LIGHT), interactive 9-layer AR mask overlays, an INCI Ingredient Safety Guard (O/X), and instant 4-language eCommerce product discovery.

What it does

• 1:1 Square Viewfinder & Ambient Light Sensor Check (CameraCaptureScreen.kt): Synchronizes CameraX Preview and ImageCapture inside a 1:1 ViewPort (UseCaseGroup) with real-time lux monitoring (Sensor.TYPE_LIGHT) so every facial capture meets YouCam Skin AI's lighting and framing requirements.
• Real-Time YouCam 9-Metric Skin AI & Interactive AR Mask Overlays (ResultScreen.kt): Calls the official Perfect Corp. YouCam Skin Analysis API (redness, oiliness, pore, acne, moisture, texture, wrinkle, age_spot, dark_circle_v2) and composites transparent PNG segmentation masks over the portrait with one-tap pills, a 6-axis Skin Balance Radar Chart, and local scan-to-scan delta tracking (▲/▼).
• Rule-Based Active Ingredient Coaching & Localized eCommerce Discovery: Deterministically maps measured skin scores to evidence-backed active ingredients (Salicylic Acid BHA 2%, Caffeine 5% + EGCG, 5D Hyaluronic Acid, Centella Asiatica, Niacinamide + Zinc) with one-tap 🛍️ Shop / Explore links localized to the user's active language (EN, JA, ZH, KO) plus an interactive INCI Ingredient Safety Guard (O/X).
• Instant 4-Language UI, Dual Clinical Themes & Routine Checklist: Zero-latency switching across English, Japanese, Simplified Chinese, and Korean, paired with Bright Clinical Pearl and Luminous Aurora themes, optional Android Health Connect sleep/hydration context, and a persistent daily Action Routine Checklist with Android reminder notifications.

How we built it

• Android Native Client: Kotlin, Jetpack Compose (Material 3), CameraX (Preview, ImageCapture, ViewPort), Android SensorManager (TYPE_LIGHT), optional AndroidX HealthConnectClient, and SharedPreferences local persistence.
• Secure FastAPI Proxy (backend/server.py): Keeps the YouCam API Secret Key strictly off the mobile APK, handles S3 pre-signed image upload and asynchronous task polling with Perfect Corp.'s YouCam Cloud, extracts mask PNGs in memory (BytesIO + Pillow), and returns structured JSON + base64 PNG masks to the Android device over an ADB reverse tunnel (tcp:8000).

Challenges we ran into

• Aligning CameraX Preview and Capture Crops: Standard Android selfie previews often differ in aspect ratio from the captured JPEG. We solved this by binding both Preview and ImageCapture to a shared 1:1 ViewPort and freezing the exact 600×600 normalized bitmap immediately upon capture.
• Multi-Language Layout Density on Mobile: Supporting English, Japanese, Chinese, and Korean without single-character line wraps required compact single-line header pills and responsive card typography.

Accomplishments that we're proud of

• End-to-end real-time integration with the official Perfect Corp. YouCam Skin AI API running on a physical Galaxy S24 (Android 14).
• Zero personal data retention: facial frames and YouCam ZIP mask archives are processed strictly in memory (BytesIO).
• Complete 4-language (EN / JA / ZH / KO) localization and dual clinical themes with localized eCommerce search links.

What we learned

• Combining computer vision skin segmentation masks from YouCam Skin AI with deterministic active-ingredient mapping and INCI safety checks transforms abstract scores into clear, high-intent eCommerce product discovery.

What's next for AuraMirror Android

• Adding longitudinal 30-day skin score trend charts and barcode/OCR cosmetic label scanning for instant on-device INCI ingredient verification.

Built With

• android
• camerax
• fastapi
• health-connect
• jetpack-compose
• kotlin
• python
• youcam-api