a more modern bunk ladder!
after!
before

Your room, reimagined.

Passthrough captures your furniture, AI redesigns it, and MR shows the upgraded 3D version exactly where it will live in real life.

Inspiration

I’ve always loved the moment when you picture your room just slightly better: a cleaner chair silhouette, a more elegant nightstand, a warmer lamp. But interior design apps always feel detached—you’re forced to browse catalog photos instead of working with your actual space.
With the new Passthrough Camera Access APIs, it suddenly became possible to do something magical: capture your real furniture directly from the Quest cameras, redesign it with AI, and place a generated 3D model back into your room in mixed reality.

AI shouldn’t just make content—it should help you reshape your actual environment.
This project is that idea, made real.

What it does

AI3D Redesign Reality is a mixed-reality furniture upgrade tool powered by Passthrough Camera Access.

Pinch to capture
Draw a bounding box around any real object—your chair, shelf, plant, lamp—and we grab that exact region from the passthrough camera feed.
AI redesigns your object
The captured snapshot is sent through an AI pipeline:

-   FLUX2 IMG

-   material + silhouette restyling

-   palette harmonization based on your actual room

Generate a 3D upgrade
Using single-image → 3D reconstruction (SAM-3D + depth priors), the redesigned object becomes a full 3D model.
Place it in mixed reality
The new object appears exactly where the old one lives, anchored in your room so you can walk around it, resize it, or swap variants.

It’s interior design without catalogs.
It’s real-time home remodeling powered by your own camera feed.
It’s reality made editable.

How we built it

Unity 6 + OpenXR Passthrough Camera Access to acquire high-resolution frames directly from the Quest’s cameras.
Custom hand-gesture bounding box tool for drawing ROI selections in 3D space.
GPU texture → CPU crop pipeline enabling precise extraction of the user-selected furniture region.
AI pipeline including:
- ControlNet for structure-preserving restyling
- Diffusion models for stylistic upgrades (minimalist, ornate, futuristic, Scandinavian, etc.)
- SAM-3D / single-image reconstruction to generate an approximate 3D mesh
MR anchoring + spatial placement so redesigned objects appear where the originals sit.
Lightweight variant switching for A/B comparisons in real time.

Challenges we ran into

Passthrough textures on Quest don’t behave like ARKit camera feeds—getting a clean, readable crop required GPU → CPU roundtripping and careful timing.
Bounding box alignment in XR is tricky without full environment meshing, so we built a camera-space projection approach.
3D reconstruction from a single image is still imperfect; we optimized it for furniture-scale silhouettes.
Lighting consistency between real and generated objects required dynamic tone matching.
Meta’s privacy throughput rules meant designing a clear, explicit user-intent workflow before capturing any camera data.

Accomplishments that we’re proud of

Built a full PCA → AI → 3D → MR pipeline that works in real time.
Designed a hand-gesture interface that feels natural, intuitive, and controller-free.
Created a workflow where users can upgrade their real furniture instantly, without scanning, meshing, or external apps.
Demonstrated a compelling practical use case for Passthrough Camera Access: real-world creative augmentation.
Turned AI design into a playful, spatial, embodied experience.

What we learned

Passthrough Camera Access is powerful, but requires careful system design to respect privacy and performance.
Users LOVE the feeling of "I just redesigned my room with my hands."
AI is most magical when grounded in the user’s real environment, not abstract prompts.
Mixed reality becomes dramatically more meaningful when it can modify the world you already live in, instead of layering fantasy on top.
Even imperfect 3D reconstructions can be extremely compelling when paired with good spatial anchoring.

What’s next for AI3D Redesign Reality

Multi-object support: redesign an entire room in one capture sweep.
On-device AI acceleration (LLAMA 3 Vision, diffusion-lite).
Higher-fidelity 3D pipelines (NeRF-style recon, SDS, textured meshes).
Scene-aware lighting estimation for even more natural MR compositing.
Marketplace of AI-generated design presets (“Japandi,” “Brutalist,” “Soft Minimalism,” “Cottagecore”).
Export redesigned furniture models to other apps or AR viewing formats.
A full “MR Home Makeover Mode” where users can try dozens of alternate room aesthetics in seconds.