Inspiration

Photogrammetry and 3D Gaussian Splatting (3DGS) represent the pinnacle of digital twin, physical AI, and spatial computing technology—allowing real-world objects and physical environments to be captured with true photorealism. However, two massive bottlenecks make 3D capture inaccessible to everyday users:

  1. The Capture Knowledge Barrier: Casual video walkthroughs shot by non-experts regularly fail in photogrammetry pipelines due to motion blur, erratic trajectory, and lighting shifts, wasting hours of compute.
  2. The "Silent Museum" Problem: Once an object or space is reconstructed in 3D, it remains a static, lifeless mesh. Users cannot interact with the geometry or ask questions about what they are looking at.

We built Splatris to turn raw casual videos into living, interactive, and embodied spatial experiences. By pairing an automated cloud photogrammetry pipeline with NVIDIA Nemotron-3-Ultra on Nebius Token Factory, Tavily Search API, and Meta Quest WebXR, Splatris creates an autonomous Spatial AI Guide that walks alongside you in virtual reality—bridging the gap between physical reality and embodied AI agents.

Quick Evaluator Note (Under 30-Second "Aha!" Moment):
To respect the judges' evaluation time, you do not need to wait for cloud GPU training. Splatris comes pre-seeded with gold-standard 3DGS models (including the Tanks and Temples M60 Tank). You can immediately orbit the 3D scene, inspect components, trigger Tavily Grounded Q&A, or jump straight into Meta Quest WebXR from your browser.


What It Does

Splatris is an end-to-end Physical AI & Spatial Intelligence platform that transforms casual walkthrough videos into interactive 3D environments guided by an embodied AI agent:

  • Automated Video Ingestion & Quality Gatekeeper: Users upload walkthrough videos (.mp4, .mov). Before burning GPU hours, an AI Gatekeeper powered by NVIDIA Nemotron-3-Ultra analyzes motion blur, camera smoothness, and frame density to approve the footage and tune COLMAP matching parameters.
  • Dual 3D Reconstruction Pipeline: Reconstructs scenes as either Standard 3DGS (for maximum photorealism on desktop) or Triangle Splats (optimized for lightweight rendering on mobile VR headsets).
  • Fast Pipeline Switch (<90s): Converts existing scenes from 3DGS to Triangle Splatting in under 90 seconds by caching and reusing COLMAP Structure-from-Motion poses.
  • Embodied Spatial AI Guide (Nemotron + Tavily): Users click or laser-point at elements in the 3D space. The system crops the corresponding camera perspective, identifies the geometry, queries Tavily Search for real-world specifications, and synthesizes an articulate response using NVIDIA Nemotron-3-Ultra.
  • Conversational Voice Narration: Streams grounded answers directly as natural speech narration through the Web Audio API with animated waveform feedback.
  • Immersive Meta Quest WebXR: Step inside the reconstruction in full 6DoF virtual reality with dedicated dual-controller ergonomics (Right Trigger = 3D laser scan; Left Trigger = HUD selection; Thumbsticks = smooth locomotion & snap turning).

How We Built It

Splatris was architected with a decoupled, high-performance tech stack across AI inference, GPU cloud compute, and immersive web visualization:

1. AI Reasoning & Video Calibration (Nebius Token Factory)

  • nvidia/Nemotron-3-Ultra-550b-a55b: We specifically leveraged the 550B parameter MoE architecture because lightweight 8B/70B models struggled with geometric reasoning. Nemotron-3-Ultra's deep chain-of-thought tokens reliably evaluate Laplacian blur distributions, cull redundant zero-parallax frames, and dynamically select COLMAP matchers (sequential vs. vocab_tree for 360° loop closure).
  • Spatial Scene RAG with Strict Guardrails: Interprets 3D user perspectives, bounding frame context, and web grounding. A system prompt guardrail ensures the guide exclusively answers questions about the active reconstructed space, declining off-topic queries.
  • Ultra-Low TTFT: Nebius Token Factory provides sub-second time-to-first-token inference, enabling natural voice narration without immersion-breaking latency.

2. Live Web Grounding (Tavily Search API)

  • The backend makes runtime calls to the Tavily Search API (search_depth: 'advanced').
  • When a user inspects an object in the 3D scene (e.g., a museum piece or military vehicle), Tavily retrieves verified production history, specifications, and cultural background, explicitly cited as Grounding: Tavily + Nemotron-3-Ultra [5 sources] to eliminate hallucinations.

3. Dedicated Cloud GPU Worker (Nebius AI Cloud)

  • Photogrammetry and differentiable splatting run on an NVIDIA® L40S GPU (48 GB GDDR6) instance on Nebius AI Cloud (bronze-bug-instance-6, Ubuntu 24.04 LTS, CUDA 13).
  • The worker executes COLMAP Structure-from-Motion and custom PyTorch rasterizers via a FastAPI microservice (splatris-worker.service).

4. Spatial Client Layer (WebXR & Three.js)

  • Built on Three.js and @mkkellogg/gaussian-splats-3d with WebGL/WebXR.
  • Features a floating glassmorphic in-VR HUD panel with dynamic layout, laser reticle projection, and Web Audio API streaming.

5. Benchmark Evaluation & Verification (Tanks and Temples)

  • To rigorously test our autonomous loop-closure calibration against gold-standard computer vision metrics, we evaluated Splatris using the renowned M60 Tank dataset from the Tanks and Temples benchmark (Knapitsch et al., SIGGRAPH 2017).

Challenges We Ran Into (Battle Scars & Engineered Solutions)

  1. Photogrammetry Fragility vs. Casual Videos (The Loop Seam Problem): Early tests with casual handheld recordings frequently produced broken point clouds or split seams at the loop closure point. We solved this by developing the Nemotron Pre-Flight Audit, which scores frames using a Laplacian variance algorithm and instructs the SfM engine to enforce visual vocabulary tree indexing (vocab_tree) whenever circular loop tours are detected.
  2. Mobile VR Splat Sorting Bottleneck: Standard 3D Gaussian Splatting requires depth-sorting hundreds of thousands of alpha-blended Gaussians per eye every frame, which quickly overpowers standalone mobile VR chipsets (Snapdragon XR2). We engineered the Triangle Splatting pipeline, generating textured primitives that run at locked 72–90 FPS in Meta Quest Browser.
  3. Audio Experience Clarity: We initially experimented with 3D positional audio emitters, but found that panning voice narration based on distance caused speech to sound muffled and uneven when users walked away from the object. We re-engineered the audio pipeline into a clear, centered Voiceover Guide with natural speech streaming, giving users an articulate tour-guide experience throughout the virtual space.
  4. TTS Length Constraints & Upstream Latency: Cloud TTS engines frequently timed out or choked on dense technical descriptions. We engineered a sentence chunker and local Web Audio buffer that streams speech immediately as tokens are decoded, maintaining voice flow without stalling.

Accomplishments That We're Proud Of

  • End-to-End Autonomous Pipeline: From raw video upload to pre-flight audit, cloud training, and interactive 3D WebXR rendering with zero command-line intervention required by the end user.
  • Sub-90-Second Fast Pipeline Switch: Caching COLMAP camera poses allows users to switch an existing scene to mobile-optimized Triangle Splats in <90 seconds without repeating the 5+ minute SfM phase.
  • Grounded Spatial Intelligence: Eliminating hallucinations by combining Nemotron's spatial reasoning with Tavily's live web retrieval.
  • Hardware-Verified Production Deployment: Documented a reproducible production setup on an NVIDIA L40S GPU on Nebius AI Cloud with verified service logs and benchmarks.
  • Gold-Standard Benchmark Verification: Successfully validated our autonomous loop-closure logic against the renowned Tanks and Temples M60 dataset, achieving clean 360° geometry without seam drift.

Benchmark Dataset Attribution

Demonstration, photogrammetry verification, and loop-closure calibrations were conducted using the standard M60 dataset from the renowned Tanks and Temples benchmark:


What We Learned

  • The Importance of Pre-Flight Quality Filtering: In compute-heavy generative 3D pipelines, an intelligent AI gatekeeper saves substantial GPU expenditure and user frustration by validating input quality upfront.
  • WebXR UX Design: Designing for VR headsets requires strict controller-hand separation (Right Hand for scene scanning, Left Hand for UI interaction) to eliminate accidental clicks and ensure smooth immersion.
  • Token Factory Speed Matters: Low-latency token generation via Nebius Token Factory is critical for conversational voice applications in spatial computing.

What's Next for Splatris

  • Multi-Agent Guided Tours: Expanding the single guide into specialized personas (e.g., an architectural historian, a structural engineer, or an art curator) who converse with each other and the user.
  • Collaborative Multi-User WebXR: Allowing multiple Quest users to walk through the same 3D Gaussian reconstruction simultaneously with synchronized laser reticles and shared audio.
  • Zero-Shot 3D Semantic Segmentation: Integrating automated open-vocabulary segmentation masks directly into the splat point cloud to isolate individual sub-components automatically.

Built With

Share this project:

Updates

Submission history