VRShop

Elevator pitch (200 characters max)

Talk to an AI interior designer while standing in your own room. It finds real products, places them at true scale in mixed reality, and buys the whole room with Visa.

About the project

Inspiration

Buying furniture online is guesswork. Will this sofa fit next to my table? Will it match my floor? Will the whole room come together, or will I end up with five pieces that each looked good on their own page?

Product photos can't answer that, and phone AR apps drop one floating item into a camera view at a time. We wanted the experience of walking into a showroom that was designed for your room: stand where you actually live, say what you want, see the real products around you at their real size, and buy all of it in one step.

What it does

VRShop is a mixed-reality shopping app for the Meta Quest 2, with an AI designer you talk to.

1. Say what you want. Hold X and speak: "Design my room as a cozy Scandinavian living room under $1,500", or "Design a Cars-themed room for my son." The designer answers out loud and plans the room around what you already own.

2. It shops real stores. Every pick is a real listing with a real price. That includes a catalog of 3,840 products from 398 stores (IKEA, Wayfair, Target, Walmart, Home Depot, Lowe's, West Elm and more), ranked by AI for your room's style, palette and free space. Themed rooms trigger a live Google Shopping search for pieces that actually carry the theme: a Lightning McQueen bed, a race-track rug, a race-car lamp.

3. It puts the room in your room. Say "put it in my room" and true-scale 3D models of those exact products appear in your space, arranged by a layout solver that knows your walls, doors and real furniture. Rugs go on the floor, table lamps and plants stand on tables, beds and sofas back onto walls, and nothing ends up inside your real couch. Pieces cast soft shadows on your real floor and disappear behind your real furniture.

4. Keep or replace what you already own. Your real furniture is kept by default. Point at your couch and say "replace this with a green velvet sofa": the real couch is painted out of the passthrough view and a product of the same kind and size stands in its spot, backed onto the same wall. Flick the thumbstick to try others.

5. Move things naturally. Drag and rotate any piece. You can drag it through real furniture while choosing a spot: it glows red and buzzes, and when you let go it glides to the nearest free spot. Walls always stop it. Lamps on a table move with the table.

6. Ask like you'd ask a person. "What's the cheapest one?", "How much is the second one?", "Anything cheaper?", "How much of my budget is left?" Numbers always come from real listing data, never invented by the model.

7. Buy the whole room with Visa. Say "buy the room" and approve once with a trigger press. An AI shopping agent then checks out at every retailer:

Every order is signed with Visa Trusted Agent Protocol (Ed25519 HTTP message signatures, single-use nonce, short expiry).
A spending mandate caps every store at its subtotal and the room at your budget.
Payments are authorized through the Visa Acceptance (Cybersource) sandbox.
An AI budget coach suggests cheaper look-alikes with one-tap swaps if you go over.
Receipts show live Visa authorization IDs. Cancelling an order is a real Visa reversal. A roommate can pay their share with a pay-by-link request.

8. Start over anytime. Say "start over" to go back to a blank design.

An optional phone companion lets you photograph your room, so the AI can read its style, palette and lighting, and review your cart. The headset works fully on its own.

How we built it

VRShop has three parts: a Quest 2 app, a Node backend, and a phone web page.

Quest 2 app (Unity 6, C#)

Meta XR SDK, MR Utility Kit and OpenXR read Space Setup: walls, floor, doors and furniture boxes, in real meters.
glTFast loads true-scale product models at runtime. Soft shadows on a shadow-catcher floor, contact shadows and depth occluders ground virtual furniture in passthrough.
A custom "room cover" shader makes a real piece disappear. For every pixel inside the piece's box, it traces the view ray to the wall or floor behind it and draws that surface at its real depth. The replacement and kept furniture then layer correctly in front of it.
An oriented-bounding-box solver (separating-axis tests) finds free spots, keeps things out of real furniture and places items on tabletops.
The whole UI, scene and materials are built in code, with no fragile prefab wiring.

Backend (Node.js, TypeScript, Express)

OpenAI does the thinking: - Room analysis and design planning with structured JSON outputs. - Ranking listings for the room. - Parsing voice intent into filters and actions (browse, place, replace, design, arrange, checkout). - Transcription and text-to-speech for the designer's voice.
Products: the IKEA catalog, plus Google Shopping via SerpAPI for every other store. Retailers come before resale marketplaces.
3D models: - IKEA's official models first (exact geometry and materials). - Then image-to-3D with fal.ai (Hyper3D Rodin), conditioned on the listing's real dimensions, for hero themed pieces. - Then a true-size stand-in textured from the product photo. - Every model is normalized with glTF-Transform, Draco and meshoptimizer into a Quest-friendly format: meters, floor pivot, decoded textures and a triangle budget.
A layout solver turns the plan plus your real room into placements that respect doors, walls and kept furniture.
Visa: - A Trusted Agent Protocol signer and our own retailer-side verifier, with a public JWKS key directory. - Mandate enforcement. - Visa Acceptance REST authorizations and reversals, with HTTP Signature auth.

Phone companion: a plain web page served by the backend for room photos, the cart and checkout.

Tests: 28 backend tests and 14 Unity EditMode tests cover the free-spot solver, stacking rules, themes, voice intent and replacement poses.

Challenges we ran into

The Quest 2 can't see your room the way you'd hope. It has no depth sensor, apps get no camera access, and passthrough is grayscale. Everything spatial had to come from Space Setup. "Removing" a real couch became a shader that reconstructs the wall and floor behind it, in grays you can match to your room.
Real products rarely come with 3D models. IKEA's official models ship in formats Unity can't load on Quest (Draco and WebP), and materials went pink in builds until we normalized every model and kept shader variants alive. For everything else we generate models from listing photos, but only when asked, because every generation costs money.
Staying on free tiers. SerpAPI gives 250 searches a month. We went catalog-first: search results, dimensions and photos are cached permanently, only successful searches count, and a themed room costs about one search per themed piece.
Virtual and real furniture colliding. Solid collisions felt frustrating, and pass-through felt fake. The answer was drag-through with a red outline and a haptic buzz, plus a snap to the nearest free spot on release.
Keeping the AI honest. The designer must never make up a price or a product. Every number it says comes from the listing data. Themes are matched in families ("Cars", "Lightning McQueen", "race car"), so the right pieces are found whichever words you use.
Hackathon networking. Venue Wi-Fi blocked device-to-device traffic, so everything runs over a phone hotspot. We also had to throttle background model downloads so they didn't starve the voice assistant.

Accomplishments that we're proud of

Standing in a real room, saying one sentence, and watching a complete, shoppable room appear around you, with real products at real sizes.
Making your real couch "disappear" and a new one take its place, on a headset with grayscale passthrough and no depth sensor.
A themed room that works for any theme: Cars, space, dinosaurs, pulled live from real stores.
A complete agentic checkout on Visa rails, verified live against the Visa Acceptance sandbox: authorized, then reversed on cancel.
A tamper demo where 10 different attacks on the agent's orders, like forged signatures, replays, expired signatures, stolen payment containers and charges over the mandate, are each rejected with a precise reason.

What we learned

In a headset, voice beats menus. The best interactions were a sentence plus a point: "replace this", "put a lamp here".
Grounding beats polygons. Shadows on the real floor and correct occlusion behind real furniture did more for realism than higher-detail models.
Agentic payments need proof, not trust. Signed requests, single-use nonces and a hard spending mandate make "let the AI buy it" something a shopper can actually agree to.
Honest fallbacks keep a demo alive. Every paid or fragile service has a clearly labelled fallback, so the whole experience runs even when a key, a network or a model is missing.

What's next for VRShop

Quest 3 support: color passthrough and the Depth API for true occlusion. The same app runs with better visuals.
Photoreal room twins from a phone scan, so replaced furniture reveals the real wall behind it.
Retailer partnerships for official 3D models, live inventory and delivery windows.
Shared rooms: co-design with a partner or roommate using shared spatial anchors, then split the bill.
Hand tracking for controller-free pinch-and-place, and streaming voice replies for an even faster designer.

Built with

unity, c#, meta-quest-2, meta-xr-sdk, mr-utility-kit, openxr, gltfast, urp, node.js, typescript, express, openai, serpapi, google-shopping, ikea, fal.ai, hyper3d-rodin, gltf-transform, draco, meshoptimizer, sharp, visa-acceptance, cybersource, visa-trusted-agent-protocol, ed25519