Reskin Surface Studio

A webXR app that redesigns any surface of any room in the way you want, and generates a full 3D Gaussian splat world for each instance - instantly viewable in the browser and VR ready in minutes.

Comment

Inspiration

Redesigning a real space is hard to imagine from a flat photo. Paint swatches and Pinterest boards don't answer the question that actually matters: what would it feel like to stand in this room if the floor were Spanish tile? We wanted to close the loop from "what if" to "I'm standing in it" - take a single photo of a real room, reskin any surface with any material, and walk through the result in VR minutes later.

What it does

Reskin Surface Studio turns one environment photo (360 or panorama) into a set of fully redesigned gaussian splat 3D worlds. Fully accessible on the web browser and webXR.

How we built it

Reskin Surface Studio (web app): Node/Express server driving Comfy Cloud's API. Per request, it generates a workflow sized to 1–3 surfaces: SAM3 mask → Gemini Nano Banana 2 masked edit → composite, chained per surface, ending in a deterministic pixel-stitched comparison sheet. Prompts are hardened automatically with a stack of guards - texture fidelity (tile grid vs. continuous material), real-world tile-size anchors calibrated against in-scene references (a door ≈ 80″), material-boundary guards, and lighting-match constraints. API keys never touch the browser.

World generation: The server uploads the final pano to World Labs and kicks off a Marble 1.1 generation, polling until the splat world is ready.

WebXR viewer: Built on IWSDK + SparkJS, and also available on the PICO Emulator. All world variants load as .spz Gaussian splats stacked at the origin; a switcher shows one at a time using Spark's native opacity. The material tray, palm-orientation detection, and pinch-grab are custom ECS systems on raw hand-joint data. The reveal effect is a GPU shader (SparkJS dyno): a spherical wavefront expands from the reveal point and splats materialize as it passes, with per-splat noise, outward push, and scale overshoot - the CPU only advances one progress uniform per frame.

Challenges we ran into

Making AI edits stay in their lane. Early generations bled texture onto adjacent walls, hallucinated new fixtures, or relit the whole room. The fix was a layered prompt-hardening stack plus CV masking: SAM3 masks intersected with user paint, mask feathering tuned per "strength" level, and hard guards on material boundaries and lighting. Tile scale realism. Models love rendering tiles at whatever size looks pretty. We had to inject real-world scale anchors with in-scene calibration references to get a 2″ mosaic to actually read as 2″. SAM3 batch quirks. SAM3 returns one mask per detection, which broke mask compositing with tensor batch-size mismatches until we flattened every mask to a union. Splats fight standard 3D assumptions. Spark ignores .visible, so world switching needed opacity-based hiding; LOD stuttered on Quest so we render splats direct; and hiding overlapping worlds at the origin took care to never double-render. Hand-tracked UX from scratch. Palm-up detection, a tray that follows your palm but stays parallel to the floor, and pinch-grab with hysteresis so it doesn't flicker - all tuned by hand until it felt natural.

Accomplishments that we're proud of

  • The full loop works live, end to end: photo → masked AI reskin → Marble splat world → walking through it in VR - in minutes, not days.
  • The palm-tray + pinch-to-apply interaction: swapping an entire world by dropping a material tile feels like magic, and the GPU ripple-reveal materializing from your hand sells it.
  • Precision surface editing that respects the real room - original lighting preserved, textures stopping exactly at material boundaries, deterministic comparison sheets that never hallucinate.
  • Five complete reskinned worlds of a real space (our hackathon lounge) switchable live on a web browser - no app install required.

What we learned

  • Prompt engineering alone can't confine an image edit - pairing generative models with classical CV masking (segmentation, mask algebra, feathering) is what makes edits precise and repeatable.
  • Gaussian splats are a genuinely viable "instant 3D" format for the browser, but they have their own rules - visibility, LOD, and animation all needed splat-native solutions.
  • Good VR interaction is 10% detection and 90% tuning: hysteresis thresholds, tray orientation constraints, and animation timing made the difference between "tech demo" and "feels right."
  • Chaining cloud AI services (Comfy Cloud → Gemini → World Labs) into one product is workable if you design for asynchrony from the start - everything polls, nothing blocks.

What's next for Reskin Surface Studio

  • Capture in headset: photograph the room from the same device you preview it in, closing the loop to a single session.
  • More surfaces and materials: beyond 3 surfaces per run, a richer material library, and user-uploaded swatch scanning.
  • In-VR editing: point at a wall in the splat world and reskin it from inside, instead of returning to the 2D studio.
  • Sharing and commerce: shareable world links for clients/roommates, and linking material tiles to real purchasable products - turning "what if" directly into a shopping list.

Built With

  • comfyui
  • pico
  • quest
  • sam3
  • tripo
  • webxr
  • worldlabs
Share this project:

Updates