Inspiration

The surest way to get clothes that fit has always been a tailor: someone who listens to what you want, takes your measurements, then cuts and sews it for you. Few people have one. Online, we guess a size from a chart, order two, and send one back, and every return costs the consumer time and the manufacturer money.

We wanted to bring the tailor into the room you're already in, with nothing to hold but your own hands, and to show a garment as real cloth on your body before anything is made.

What it does

Dress Room is a tailor in your room: an agentic, hands-only mixed reality atelier for Meta Quest 3. You say what you want; an AI agent decides how to make it and does the work with the app's own tools. It is not a wrapper around a 3D or image generator.

  • An agent that runs the atelier. Wondering what to wear next, or what to buy? Make a fist and your hand becomes the microphone. Say what you have in mind, a dress for a friend's wedding or a jacket for work, then open your hand to send, and try it on here before you buy anything. From that one sentence, the agent chooses what to do:
    • draft a new garment on your measurements: "a knee-length red dress";
    • redraft its pattern: "make it shorter, with long sleeves";
    • dress you for an occasion: "dress her for a party in Paris in 2026";
    • change the fabric, print, colour or light: "give it golden hour light";
    • take you back, with intuitive version control: "actually, I preferred the dress".

It keeps what you're already wearing and knows which garments can be worn together. When something can't be made, it says so and makes the nearest thing. What it drafts is a real sewing pattern, not a picture.

  • Fit it to you. Pinch the tape measures on the avatar and pull them to your size. Every garment is then drafted again on that body.
  • Watch it being made. The headset cuts the pattern pieces, sews them along their seams and settles the garment on the body in about 8–15 seconds, then keeps the cloth alive. Push the skirt, pinch the fabric, pull it and let go.
  • Everything by hand, seated. A glance at your wrist opens a ring of six controls: the collection, rotate, bring the avatar to you, settings, your room or immersive, and your measurements. Point at an arch in the gallery and pinch to wear one of about 250 looks from our collection. The other wrist keeps every version of the session: poke one and it's back in under a second. Your palm offers fabrics.
  • In your room. In passthrough, Dress Room reads the room layout from the headset's scene data to put the gallery's arches on a wall and the avatar in a free spot, and anchors them so they stay where they are.

A short walk-through teaches the gestures at launch.

How we built it

The app. Unity 6 (URP, Vulkan) with the Meta XR Core and Interaction SDK v207, hand tracking only. The wrist controls use poke; the gallery and the garment pieces use pointing rays from the hand's pointer pose with a Burst-compiled raycast. The fist gesture has measured thresholds: the palm raised to within 22 cm below the eyes and held for 0.3 s, so it doesn't fire while you gesture.

Our drape engine. We wrote the cloth simulation for the Quest GPU from scratch. It is XPBD (extended position-based dynamics) in compute shaders. For each constraint $C_j$ with compliance $\alpha_j$, each substep solves

$$ \Delta\lambda_j = \frac{-C_j(\mathbf{x}) - \tilde{\alpha}_j\,\lambda_j}{\nabla C_j\,\mathbf{M}^{-1}\,\nabla C_j^{\top} + \tilde{\alpha}_j}, \qquad \Delta\mathbf{x} = \mathbf{M}^{-1}\,\nabla C_j^{\top}\,\Delta\lambda_j, \qquad \tilde{\alpha}_j = \frac{\alpha_j}{\Delta t^{2}} $$

Constraints are graph-coloured. Every colour is one dispatch across the whole GPU, and colours with few constraints share a single workgroup. The body is a signed distance field, and the hands are spheres that push and pinch the cloth. Cloth-against-cloth collision runs only near the hands, where you would see it. The mesh is adaptive: 1 cm where cloth folds and 2 cm elsewhere. The renderer adds Phong tessellation and occlusion from the particles, and nothing is read back to the CPU each frame.

Sewing on the headset. A pattern arrives from our backend, which runs an extended version of GarmentCode, as 2D pieces and their seams. The Quest triangulates the pieces, sews them around the body and settles them on four low-priority CPU threads, so the frame rate holds while a garment is being made. A short GPU pre-settle then runs out of sight.

The backend and the agent. A FastAPI service hosts the atelier agent: Claude, working through tools.

  • make_garment drafts with GarmentCode (open source, MIT) on the wearer's measurements, from about 30 garment types: dresses, skirts, trousers, jumpsuits, shirts and tailored jackets with lapels, pockets and pleats.
  • make_look wears a look from the collection.
  • set_look changes colour, 19 scanned fabrics, 26 prints, sheen, the mannequin's finish and the light.
  • Versions let you undo or go back.

Speech is transcribed by NVIDIA's Nemotron model through OpenRouter. The model only listens and decides. The tools hold the craft: which garments can be worn together, how necklines and layers behave, and what to keep when a garment is redrafted. On our 50-request benchmark, every graded trial in the last full run passed (72 of 72).

Cheap to run, by design. The AI only listens and decides. Drafting is GarmentCode, with no model call, and the cutting, sewing and simulation run on the headset itself, not on our servers. That keeps every request cheap to serve, and fewer returns lower the costs for consumers and manufacturers.

Testing. We drive synthetic hands on the real headset to press the real buttons, and measure every change on the device with VrApi and Perfetto.

Challenges we ran into

  • The frame budget. At 72 Hz, passthrough, rendering and simulation share $\tfrac{1}{72\,\mathrm{Hz}} \approx 13.9\ \mathrm{ms}$. A 1 cm dress of 20,868 particles first ran at 38 fps (24.4 ms). Moving the solver from a single workgroup to dispatches across the whole GPU brought it to 73 fps (10.9 ms). These numbers are on the cable. On battery, passthrough holds the GPU at 456 MHz instead of 640 MHz, so battery-speed tuning is part of what's next.
  • Sewing without a hitch. Making a garment is heavy work. It runs at low priority across several threads so the frame rate holds while you watch.
  • Cloth against cloth. Full self-collision is too costly on a mobile GPU, so we run it only where your hands are.
  • Layers. A loose top pokes through a jacket. The agent now makes the layer underneath fitted, and an outfit is capped at three garments.
  • Gestures that don't misfire. A fist is a natural thing to make, so the mic waits for a raised palm and a short hold, and it shows over your hand while it listens.
  • Waiting. An agent turn plus sewing takes 20–40 seconds for a new garment. A making card shows each step, and one request at a time keeps it clear.
  • Changing a garment without sewing it again. We tried live edits, and parked them until they hold up.

Accomplishments that we're proud of

  • A fully agentic workflow: one spoken sentence becomes a drafted, sewn and simulated garment, with no menus in between.
  • Real garments cut, sewn and simulated live on a standalone headset, with your hands in the cloth, at 72 fps (measured on the cable).
  • Fully hands-only, from first launch to the last look: no controller is ever needed.
  • An agent that drafts real sewing patterns. Take the AI away, and the drafting, sewing, simulation and collection still work.
  • From the first commit on 1 November to release builds on a Meta test channel in under three weeks.

What we learned

  • On a mobile GPU, how work is dispatched matters more than the maths inside it.
  • An agent is only as good as its tools. Put the tailoring knowledge in the tools and let the model choose.
  • Hands-only design needs one clear gesture per meaning, and feedback where your eyes already are.
  • Measure on the device, at battery speed, and never assume.

What's next for Dress Room

  • Ready to buy: find garments like the one you designed.
  • Send it to a maker: hand the pattern, drafted to your measurements, to a manufacturer, so it fits the first time, cutting return costs for consumers and manufacturers.
  • Edit without re-sewing, so small changes are instant.
  • Measurements from a quick scan instead of tape measures.
  • More rooms, more speed: room placement tested in many homes, and full-speed simulation on battery.
  • More languages.
  • Target launch: the Meta Horizon Store, in 2027.

Built With

Share this project:

Updates

Submission history