Splatris

Convert videos into photorealistic 3D Gaussian & Triangle Splats with an embodied Spatial AI Guide powered by NVIDIA Nemotron on Nebius Token Factory and Tavily web grounding in WebXR.

Comment

Inspiration

Photogrammetry and 3D Gaussian Splatting (3DGS) represent the pinnacle of digital twin and spatial computing technology—allowing real-world objects and physical environments to be captured with true photorealism. However, two massive bottlenecks make 3D capture inaccessible to everyday users:

  1. The Capture Knowledge Barrier: Casual video walkthroughs shot by non-experts regularly fail in photogrammetry pipelines due to motion blur, erratic trajectory, and lighting shifts, wasting hours of compute.
  2. The "Silent Museum" Problem: Once an object or space is reconstructed in 3D, it remains a static, lifeless mesh. Users cannot interact with the geometry or ask questions about what they are looking at.

We built Splatris to turn 3D Gaussian Splats into living, interactive, and embodied spatial experiences. By pairing an automated cloud photogrammetry pipeline with NVIDIA Nemotron-3-Ultra on Nebius Token Factory, Tavily Search API, and Meta Quest WebXR, Splatris creates an autonomous Spatial AI Guide that walks alongside you in virtual reality.

What It Does

Splatris is a full-stack spatial intelligence platform that transforms raw walkthrough videos into interactive 3D environments guided by an embodied AI agent:

  • Automated Video Ingestion & Quality Gatekeeper: Users upload walkthrough videos (.mp4, .mov). Before burning GPU hours, an AI Gatekeeper powered by NVIDIA Nemotron-3-Ultra analyzes motion blur, camera smoothness, and frame density to approve the footage and tune COLMAP matching parameters.
  • Dual 3D Reconstruction Pipeline: Reconstructs scenes as either Standard 3DGS (for maximum photorealism on desktop) or Triangle Splats (optimized for lightweight rendering on mobile VR headsets).
  • Fast Pipeline Switch (<90s): Converts existing scenes from 3DGS to Triangle Splatting in under 90 seconds by caching and reusing COLMAP Structure-from-Motion poses.
  • Embodied Spatial AI Guide (Nemotron + Tavily): Users click or laser-point at elements in the 3D space. The system crops the corresponding camera perspective, identifies the geometry, queries Tavily Search for real-world specifications, and synthesizes an articulate response using NVIDIA Nemotron-3-Ultra.
  • Conversational Voice Narration: Streams grounded answers directly as natural speech narration through the Web Audio API.
  • Immersive Meta Quest WebXR: Step inside the reconstruction in full 6DoF virtual reality with dedicated dual-controller roles (Right Trigger = 3D laser scan; Left Trigger = HUD selection; Thumbsticks = smooth locomotion & snap turning).

How We Built It

Splatris was architected with a decoupled, high-performance tech stack across AI inference, GPU cloud compute, and immersive web visualization:

1. AI Reasoning & Video Calibration (Nebius Token Factory)

  • nvidia/Nemotron-3-Ultra-550b-a55b: Evaluates Laplacian blur metrics and frame coverage during the pre-flight audit, dynamically selecting COLMAP matchers (sequential vs. vocab_tree) and densification iterations.
  • Spatial Scene RAG with Strict Guardrails: Interprets 3D user perspectives, bounding frame context, and web grounding. A system prompt guardrail ensures the guide exclusively answers questions about the active reconstructed space, declining off-topic queries.
  • Ultra-Low TTFT: Nebius Token Factory provides lightning-fast time-to-first-token inference, enabling natural voice narration without immersion-breaking latency.

2. Live Web Grounding (Tavily Search API)

  • The backend makes runtime calls to the Tavily Search API (search_depth: 'advanced').
  • When a user inspects an object in the 3D scene (e.g., a museum piece or military vehicle), Tavily retrieves verified production history, specifications, and cultural background, preventing hallucinations.

3. Dedicated Cloud GPU Worker (Nebius AI Cloud)

  • Photogrammetry and differentiable splatting run on an NVIDIA® L40S GPU (48 GB GDDR6) instance on Nebius AI Cloud (bronze-bug-instance-6, Ubuntu 24.04 LTS).
  • The worker executes COLMAP Structure-from-Motion and custom PyTorch rasterizers via a FastAPI microservice (splatris-worker.service).

4. Spatial Client Layer (WebXR & Three.js)

  • Built on Three.js and @mkkellogg/gaussian-splats-3d with WebGL/WebXR.
  • Features a floating glassmorphic in-VR HUD panel with dynamic layout, laser reticle projection, and Web Audio API streaming.

Challenges We Ran Into

  1. Photogrammetry Fragility vs. Casual Videos: Early tests with casual phone and headset recordings frequently produced broken point clouds due to motion blur and missing overlap. We solved this by developing the Nemotron Pre-Flight Audit, which scores frames using a Laplacian variance algorithm and prompts users with actionable capture advice before dedicating GPU time.
  2. Mobile VR Splat Sorting Bottleneck: Standard 3D Gaussian Splatting requires sorting hundreds of thousands of alpha-blended Gaussians per eye every frame, which chokes standalone mobile VR chipsets (Snapdragon XR2). We implemented the Triangle Splatting pipeline, generating textured primitives that run at locked 72–90 FPS in Meta Quest Browser.
  3. Audio Experience Clarity: We initially experimented with 3D positional audio emitters, but found that panning voice narration based on distance caused speech to sound muffled and uneven when users walked away from the object. We re-engineered the audio pipeline into a clear, centered Voiceover Guide with natural speech streaming, giving users an articulate tour-guide experience throughout the virtual space.

Accomplishments That We're Proud Of

  • End-to-End Autonomous Pipeline: From raw video upload to pre-flight audit, cloud training, and interactive 3D WebXR rendering with zero command-line intervention required by the end user.
  • Sub-90-Second Fast Pipeline Switch: Caching COLMAP camera poses allows users to switch an existing scene to mobile-optimized Triangle Splats in <90 seconds without repeating the 5+ minute SfM phase.
  • Grounded Spatial Intelligence: Eliminating hallucinations by combining Nemotron's spatial reasoning with Tavily's live web retrieval.
  • Hardware-Verified Production Deployment: Documented a reproducible production setup on an NVIDIA L40S GPU on Nebius AI Cloud with verified service logs and benchmarks.

What We Learned

  • The Importance of Pre-Flight Quality Filtering: In compute-heavy generative 3D pipelines, an intelligent AI gatekeeper saves substantial GPU expenditure and user frustration by validating input quality upfront.
  • WebXR UX Design: Designing for VR headsets requires strict controller-hand separation (Right Hand for scene scanning, Left Hand for UI interaction) to eliminate accidental clicks and ensure smooth immersion.
  • Token Factory Speed Matters: Low-latency token generation via Nebius Token Factory is critical for conversational voice applications in spatial computing.

What's Next for Splatris

  • Multi-Agent Guided Tours: Expanding the single guide into specialized personas (e.g., an architectural historian, a structural engineer, or an art curator) who converse with each other and the user.
  • Collaborative Multi-User WebXR: Allowing multiple Quest users to walk through the same 3D Gaussian reconstruction simultaneously with synchronized laser reticles and shared audio.
  • Zero-Shot 3D Semantic Segmentation: Integrating automated open-vocabulary segmentation masks directly into the splat point cloud to isolate individual sub-components automatically.

Built With

Share this project:

Updates

Submission history