SAM Edits

Splat editing with smart objects

Comment

Inspiration

  • Gaussian splats capture the real world photorealistically,but they're frozen: you can look, never touch.
  • We wanted every object in a scanned room to be individually understood, measured, and movable.
  • One capture becomes a restageable digital twin, useful for virtual production, real-estate staging, retail, and embodied-AI training, building agent first game environments.

What it does

  • Takes a 3D splat of a real room and uses SAM2 + a vision-language model to break it into separate, labeled, measured objects.
  • Lets you redecorate by talking: "put the lamp on the coffee table" an LLM parses it, a spatial resolver places the object respecting walls, floors, and collisions.
  • A vision model then looks at the result, validates the placement, and auto-corrects it — and you can walk through the room first-person on a PICO headset emulator while a teammate live-edits it.

How we built it

  • Frontend: React Three Fiber web app with orbit, first-person walk, and PICO emulator support; backend: FastAPI.
  • Reconstruction: SAM2 segmentation + VLM labeling + FLUX completion to lift objects out of splats; placement runs on a pure-geometry resolver with room-aware bounds measured by raycasting the actual wall surfaces.
  • All AI runs locally: llama3.2 parses commands, Qwen2.5-VL visually validates placements (both via Ollama), with a command-relay so any device can direct any view.

Challenges we ran into

  • No clear Splat data to train on, we wanted an HD splat to run through our workflow but we couldn't find any splat that had enough resolution for SAM object detection
  • Making tiny local models reliable took careful few-shot prompt work "add a plant" parsed wrong until the examples taught it.

Accomplishments that we're proud of

  • A complete loop: real capture → segmented objects → natural-language restaging → self-validating placements.
  • The core loop needs zero cloud AI, everything runs on one laptop.

What we learned

  • Room geometry must be measured, not assumed, the difference between a bounding box and the real interior is a table stuck in a wall.
  • Small local models are enough when the prompt does the heavy lifting.
  • VR platforms are uneven, design every feature with a graceful fallback.

What's next for SAM Edits

  • True immersive WebXR on real PICO hardware, stand inside the room while it restages around you.
  • Add textures and lighting so you can truely experience what the environment feels like!

Built With

Share this project:

Updates