Boo, Darling!

Boo, Darling! is a hands-free AI costume mirror. Stand in front of the mirror, invoke the Baroness, and receive an Aura Score of your fashion sense and a witty spoken critique of your outfit.

Comment

Boo, Darling!

Inspiration

We wanted to make costume feedback feel like an experience, not a questionnaire.

Inspired by Cruella's dramatic fashion sense and her ability to make every outfit feel like a performance, Boo, Darling! transforms an ordinary webcam into a haunted fashion mirror where the Baroness decides who is fairest of them all.

We also wanted the experience to be completely hands-free. No uploading photos, navigating menus, or touching a keyboard. You step in front of the mirror, say the magic words, and let the Baroness take over.

What it does

Boo, Darling! is a hands-free AI costume mirror.

Once the camera is enabled, the mirror:

  • Detects when someone enters the camera frame.
  • Uses body landmarks to evaluate positioning and framing.
  • Waits for a stable pose before capturing.
  • Samples multiple frames and selects the sharpest image.
  • Automatically crops the outfit for visual analysis.
  • Compares the outfit against curated Cruella-inspired reference images.
  • Generates an Aura Score out of 100.
  • Reveals the closest matching reference look.
  • Generates a theatrical, personalized critique from the Baroness.
  • Speaks the critique aloud using ElevenLabs.
  • Automatically resets for the next visitor.

The Aura Score measures visual resemblance to the reference outfit, not personal attractiveness or overall fashion quality.

How we built it

Boo, Darling! combines real-time computer vision, vision-language models, embedding-based image retrieval, and speech synthesis into a multi-stage inference pipeline.

1. Real-time computer vision and intelligent frame selection

We built a custom capture controller using MediaPipe Pose Landmarker (@mediapipe/tasks-vision 1.2.0) to handle real-time person detection, body landmark extraction, and automatic framing.

Rather than capturing the first available webcam frame, our controller uses temporal stability checks to determine when a visitor is properly positioned. It tolerates transient tracking failures, waits for positional stability, and samples multiple candidate frames over a short capture window.

To select the best frame, we use Variance of the Laplacian, a classical computer vision technique for estimating image sharpness.

The Laplacian operator highlights rapid intensity changes, such as edges and fine details. Blurry images generally produce lower variance, while sharper images produce higher variance.

We select the candidate with the highest sharpness score instead of relying on an arbitrary snapshot.

After selection, pose landmarks around the shoulders, hips, and lower body help construct a padded clothing-region crop. The image preserves its aspect ratio and is encoded as a JPEG data URL, with a maximum decoded payload size of 1 MB.

This combines semantic spatial information from pose estimation with a lightweight classical image-quality metric to improve the quality of downstream model inputs.

2. FashionCLIP and embedding-based image retrieval

We built a separate Python similarity service using FashionCLIP, with a standard CLIP image encoder as a fallback.

Instead of asking an LLM whether a costume looks like Cruella, we formulate costume matching as an image-to-image retrieval problem.

The pipeline works as follows:

  1. Encode the visitor's cropped outfit using a pretrained image encoder.
  2. L2-normalize the resulting feature vector to produce a unit-length embedding.
  3. Compare it against precomputed reference embeddings using cosine similarity.
  4. Aggregate similarity scores across images belonging to the same reference look.
  5. Select the look with the highest average similarity.
  6. Retrieve the individual reference image with the highest similarity for the final reveal.

Mathematically, for normalized embeddings:

We initially curated three distinct reference images for our Cruella look, with each image belonging to a specific lookId.

Reference embeddings are precomputed and cached rather than repeatedly encoding the same reference images for every visitor. This reduces redundant inference and keeps the live comparison pipeline lightweight.

Importantly, we use a pretrained encoder without additional training or fine-tuning.

3. Similarity calibration and the Aura Score

One challenge with embedding-based retrieval is interpreting the output.

A cosine similarity of 0.76 does not mean that a costume is 76% accurate. Embedding similarity is a relative measure of visual feature alignment, not a probability or percentage of correctness.

We therefore introduced a fixed calibration layer that maps the winning look similarity to a 0–100 Aura Score.

The anchors are selected using unrelated, partial-match, and strong-match outfit examples processed through the same reference pipeline.

We also track three versions alongside every successful result:

  • modelVersion: identifies the embedding encoder.
  • referenceVersion: identifies the curated reference set.
  • calibrationVersion: identifies the scoring parameters.

This prevents changes to the encoder or reference collection from silently reusing incompatible calibration values.

The result is a deterministic, reproducible game score rather than an LLM-generated numerical judgment.

4. Separating visual evaluation from generative reasoning

We deliberately separated the scoring engine from the Baroness.

Our system has two distinct AI responsibilities:

Visual evaluator: FashionCLIP computes image embeddings, retrieves the closest reference look, and determines the Aura Score.

Generative critic: A separate vision-language model receives both the visitor's outfit and the selected reference image, along with the computed result. It generates a theatrical critique describing visible styling differences and suggesting improvements.

The critic cannot modify the numerical score.

This separation is important because generative models can produce inconsistent numerical judgments. By keeping the evaluator deterministic and the critic generative, we can provide creative, contextual feedback without allowing generated text to manipulate the underlying result.

We also validate generated responses and provide preset critiques when the vision-language service fails or times out. The system retains the valid similarity result even if the critic becomes unavailable.

5. Session-aware asynchronous orchestration

The frontend is built using Next.js 14.2.15, React 18.3.1, and TypeScript 5.5.4.

We use a state-driven interaction flow to coordinate multiple asynchronous processes:

WAITING → FRAMING → CAPTURING → ANALYZING → RESULTS → RESET

Each visitor receives a unique sessionId, which is propagated through the capture, scoring, critique, and speech pipelines.

This is particularly important because camera tracking, model inference, and speech synthesis have different execution times.

Our orchestration handles:

  • Debounced presence detection to avoid unnecessary session transitions.
  • Transient tracking loss without immediately restarting capture.
  • One assessment in flight at a time.
  • Request cancellation when a visitor leaves.
  • Stale-response rejection to prevent previous visitors' results from appearing in new sessions.
  • Timeouts and graceful fallbacks for unavailable services.
  • Audio cancellation and automatic session cleanup.

The frontend communicates with separate server-side API routes for rating, critique generation, and speech synthesis. The Python similarity service is kept behind the server-side interface rather than exposing model infrastructure directly to the browser.

6. Voice synthesis and theatrical presentation

We use ElevenLabs Text-to-Speech to turn the Baroness's critique into spoken feedback.

The voice layer is deliberately independent of the evaluation pipeline. It receives the generated critique and handles speech synthesis without influencing the score.

Challenges we ran into

Making computer vision reliable in an uncontrolled environment.

A webcam introduces motion blur, inconsistent positioning, partial visibility, and changing lighting conditions. We learned that person detection alone does not guarantee a usable image. We needed temporal stability checks, landmark-based framing, and sharpness-based frame selection to improve input quality before inference.

Interpreting embedding similarity correctly.

Cosine similarity is not equivalent to classification confidence or costume accuracy. We had to introduce a separate calibration layer and ensure that our scoring parameters remained consistent with the encoder and reference dataset.

Coordinating multiple asynchronous services.

Camera input, pose estimation, image capture, Python inference, generative critique, and speech synthesis all operate on different timelines. We had to handle cancellation, timeouts, session synchronization, and service failures without leaving the mirror in a broken state.

Balancing generative flexibility with deterministic evaluation.

We wanted the Baroness to have personality without letting an LLM arbitrarily decide someone's score. Separating visual retrieval from generative critique gave us a much clearer boundary between measurable similarity and subjective commentary.

Accomplishments that we're proud of

We are particularly proud of turning several independent technologies into one cohesive, hands-free interaction.

Technically, some of our favourite decisions were combining MediaPipe pose estimation with Laplacian-based sharpness scoring, implementing multi-reference embedding retrieval, and building a version-aware calibration layer for our Aura Score.

We also designed a clear separation between deterministic evaluation and generative feedback. The Baroness can be dramatic, opinionated, and entertaining, but she cannot manipulate the verdict.

What we learned

We learned that building an AI-powered experience is as much about controlling inputs and defining system boundaries as it is about choosing models.

A particularly important lesson was that different models should have clearly defined responsibilities. The embedding model measures visual resemblance, the vision-language model explains visible differences, and the speech model creates atmosphere.

We also discovered how relevant classical computer vision remains in modern AI applications. A relatively inexpensive image sharpness metric, combined with pose estimation and temporal filtering, can meaningfully improve the quality of inputs before they reach a much larger model.

What's next for Boo, Darling!

We want to expand beyond one Cruella-inspired look and build a larger library of character outfits with properly curated reference images.

We would also like to:

  • Improve person detection and multi-person handling.
  • Introduce clothing segmentation to isolate garments more precisely and reduce background interference.
  • Experiment with different embedding models and evaluate retrieval performance across diverse costume examples.
  • Improve calibration using a larger collection of real costume comparisons.
  • Add shareable results cards.
  • Support more voice styles and characters.
  • Introduce a second phase using MediaPipe Face Landmarker for facial-expression tracking and a separate Scare Score.
  • Improve operator readiness checks and test across different cameras, lighting conditions, and backgrounds.

Built With

Share this project:

Updates

Submission history