Vogue Vision

Instant fashion verdicts powered by AI

Comment

Inspiration

Fashion feedback is usually static: a comment, a rating, a photo review. What if fashion feedback was real-time, visual, and live?

Vogue Vision turns getting dressed into a runway moment. Point your camera at yourself and the mirror reads your outfit—not your face. Gemini looks at what you're actually wearing, infers the occasion, and writes you a verdict: a name for the look, a reaction, a styling tip, and a score. Then it directs a live AR scene around your body to match.

Wearing something bold and metallic? You get prisms and a disco scene. Soft florals? Butterflies and wings. The AI picks the effects, the colors, and where on your body they attach.

The centerpiece is the typography: Gemini names your look, and that name materializes as giant 3D balloon letters standing behind you—not a sticker pasted on the video, but actual depth. Your body passes in front of them the way a physical object would. That depth comes from real-time person segmentation, creating true occlusion.

And it's live. The whole thing runs through a Vonage video session from the first click. Send someone the same link and they watch your AR mirror in real time, on the same page, no separate viewer app.

The insight: Fashion feedback is social and it's live. A static filter is a photo. This is a runway you can broadcast.

How We Built It

AI: Gemini 3.6 Flash with Structured Output

Gemini analyzes the outfit and returns a strict JSON contract—outfit context, color analysis, fit observations, style elements, vibe description, styling tips, score. We use responseJsonSchema to enforce structure, and a Zod schema validates every response server-side before it reaches the renderer. A malformed or adversarial output can never drive the graphics. The API key never leaves the server.

Video: Vonage Video API

  • @vonage/server-sdk mints sessions and publisher JWTs
  • @vonage/client-sdk-video runs in the browser
  • Vonage is load-bearing, not bolted on: it creates the session first, acquires the camera itself, and everything downstream reads frames out of that stream

Computer Vision: MediaPipe Tasks Vision

Runs in a Web Worker so tracking never blocks the render loop. Provides 33 pose landmarks plus a per-pixel person segmentation mask at ~12fps.

Graphics: Three.js with Custom GLSL

The segmentation mask feeds a three-pass renderer:

  1. Geometry behind the person
  2. Masked person cutout
  3. Geometry in front

This is what makes occlusion real. Effects are scaled to shoulder span, so they track distance from the camera automatically.

Backend: Node.js + Express

  • Vite 7 + vanilla JS frontend
  • Node 22 + Express 5 backend
  • No framework overhead
  • One server, one port, one URL for broadcaster and viewer

Challenges We Faced

Real-Time Segmentation + 3D Rendering

MediaPipe + Three.js in the browser could easily drop frames. Solution: run MediaPipe in a Web Worker (off main thread), cache segmentation masks, use WebGL for GPU-accelerated rendering. Result: consistent 30fps.

True 3D Occlusion from 2D Mask

Segmentation is a 2D mask. We needed to convert it to true 3D occlusion. Solution: three-pass render pipeline with depth testing. The mask becomes a stencil, controlling what's drawn in front vs. behind. Now AR objects occlude naturally.

AI Output Reliability

Gemini needed to return structured data every time, not prose. Solution: responseJsonSchema enforces JSON structure, Zod validates on the server. No parsing errors, no hallucinations reaching the renderer.

Live Broadcast Without Latency

Both broadcaster and viewer on the same session, seeing the AR in real time. Solution: Vonage handles the heavy lifting; we just read frames from the stream and render. Latency is sub-second.

Effect Positioning to Body

Effects need to follow your movement in 3D space. Solution: Map MediaPipe's 2D keypoints to 3D coordinates using perspective projection. Effects scale as $$scale = \frac{shoulder_span}{reference_width}$$. Rotation uses pose angles (hip-shoulder vector). Now confetti moves with you, wings follow your arms.

What We Learned

  1. MediaPipe + Web Workers = smooth real-time ML: Off-loading pose detection prevents frame drops
  2. Structured outputs prevent hallucinations: Zod schemas validate before anything reaches the frontend
  3. Three-pass rendering handles occlusion: True depth-based occlusion beats alpha blending
  4. Vonage is solid for real-time: Once you get auth right, the video experience is rock-solid
  5. Testing AI systems requires defensive validation: We test prompt injection, malformed outputs, rate limits, concurrent requests
  6. One backend, one frontend, one URL scales better: No separate viewer app—just share the link

Results

  • ✅ Live AR fashion mirror running at 30fps
  • ✅ Real-time person segmentation with true 3D occlusion
  • ✅ Gemini-powered outfit analysis with structured output validation
  • ✅ Vonage broadcast from one link to multiple viewers
  • ✅ 11 tests covering API validation, prompt injection, malformed outputs, and coordinate mapping

One-Liner

Gemini styles you, Vonage broadcasts you, and the AR stands behind you in 3D.

Built for Runway to Reality: AI Fashion Hackathon (Google Developer Groups + Vonage, LUMA Studios NYC).

Built With

Share this project:

Updates

Submission history