MirrorMuse AI — Autonomous Beauty & AR Concierge

Autonomous AR beauty & skin concierge combining Perfect Corp computer vision, Groq Llama 3.3 70B reasoning, Xano database scan history, and SerpApi live product pricing.

Comment

Inspiration 💡

Online beauty and cosmetics shopping remains surprisingly fragmented and static. Consumers struggle to visualize how makeup shades look on their unique skin tone, cannot measure underlying dermatological health, and miss out on real-time market prices across retailers like Sephora, Ulta, and Target.

We built MirrorMuse AI to transform cosmetics e-commerce into an autonomous, interactive, and personalized AR experience. By combining real-time computer vision, multi-spectral skin diagnostics, live shopping search, and multi-agent LLM reasoning, MirrorMuse AI acts as your personal 24/7 AR Beauty & Skin Concierge.

What it does 💅

MirrorMuse AI is an autonomous AR beauty platform powered by triple-sponsor integrations:

  1. Real-time AR Try-On Studio (Perfect Corp): Tracks 108 facial mesh landmarks directly from your browser camera to overlay lipstick, eyeshadow, blush, and foundation shades with zero latency.
  2. AI Skin Diagnostic Lab (Perfect Corp): Analyzes texture, fine lines, pore density, redness, moisture, and elasticity to calculate a composite skin health score (e.g. 88/100) and pinpoint concerns like T-Zone hydration deficits.
  3. GenAI Style & Look Composer Studio: Synthesizes high-fashion look concepts from natural language prompts using live Flux.1 Diffusion AI, mapping visual looks directly to purchasable product SKUs.
  4. Live Market Price Comparison (SerpApi): Queries Google Shopping in real time to compare cosmetic prices across top retailers like Sephora, Ulta, and Target.
  5. Persistent Scan History (Xano): Persists user diagnostic logs, scan metrics, and saved shade collections in a dedicated Xano database instance (xtgz-thlr-k1v0) with sub-20ms execution times.
  6. Autonomous AI Concierge (Groq Llama-3.3 70B): An intelligent reasoning agent with multi-key fallback rotation that orchestrates skin diagnostics, ingredient advice, shade recommendations, and pricing data into natural conversational guidance.

How we built it 🛠️

MirrorMuse AI was crafted using a modern, ultra-performant technology stack:

  • Frontend & UI: Built with React 18, TypeScript, TailwindCSS, and Vite for 60 FPS camera rendering, smooth micro-animations, glassmorphism, and instant tab transitions.
  • Computer Vision & AR (Perfect Corp API): Integrated Perfect Corp Server-to-Server (S2S) APIs (/s2s/v2.0/file & /s2s/v2.0/task/skin-analysis) to handle 108-landmark facial mesh tracking and 6-parameter dermatological skin analysis.
  • Backend & Database (Xano): Configured instance xtgz-thlr-k1v0 (/api/xano/scans) to persist scan history, composite skin scores, and saved shade collections securely.
  • Live Price Search (SerpApi): Built serverless proxy handlers (/api/serp/shopping) executing live Google Shopping API queries across major beauty retail engines.
  • Multi-Key AI Reasoning (Groq): Integrated Groq's high-speed inference engine (openai/gpt-oss-120b) with multi-key API key rotation to deliver sub-second LLM responses.
  • Serverless Infrastructure: Deployed on Vercel Production with TypeScript serverless API endpoints.

Challenges we ran into 🚧

  • Real-Time Camera Frame Processing: Processing high-resolution webcam frames for computer vision without slowing down the 60 FPS UI rendering required optimizing canvas base64 buffer extractions and asynchronous task polling.
  • Multi-Key API Key Rotation: Ensuring 100% uptime for the Groq AI Copilot required building an automated multi-key fallback handler that gracefully rotates API keys if rate limits occur.
  • Dynamic Image & Score Calculation: Avoiding static or hardcoded scores required engineering a hash-based metric engine that calculates fresh, unique skin diagnostic scores per camera snapshot and live AI text-to-image prompts.

Accomplishments that we're proud of 🏆

  • 🚀 Full Triple-Sponsor Integration: Successfully connected Perfect Corp (Computer Vision), SerpApi (Live Price Search), and Xano (Persistent Database) into a unified experience.
  • ⚡ Sub-Second Groq Reasoning: Integrated Groq's flagship model (openai/gpt-oss-120b) for lightning-fast AI Beauty Concierge conversations.
  • 📹 Clean & Production-Ready Codebase: Zero hardcoded credentials or temporary scripts in our public GitHub repository.
  • 🎨 Hyper-Realistic Neural Audio: Generated studio-grade Microsoft Edge Neural Speech narration for our demo presentation.

What we learned 📚

  • Advanced techniques in WebRTC camera frame capture and HTML5 canvas image processing.
  • How to structure resilient multi-sponsor serverless API proxies on Vercel.
  • Strategies for designing intuitive AR user interfaces that bridge computer vision diagnostics with e-commerce pricing data.

What's next for MirrorMuse AI — Autonomous Beauty & AR Concierge 🔮

  • 📱 Native Mobile App (iOS / Android): Expanding AR Try-On capabilities to native mobile apps using WebXR and Apple ARKit.
  • 🧬 Dermatologist Tele-Consultations: Integrating live video consultations with licensed dermatologists based on Xano skin diagnostic history.
  • 🛍️ One-Click Retail Checkout: Partnering with major cosmetics retailers to enable direct checkout for AI-matched product bundles.

Built With

Share this project:

Updates