ChronoSplat 4D

Hands-first 6DoF volumetric cinema studio for Meta Quest 3 with 90 FPS 4D Gaussian Splatting and Dual-LLM spatial copilot.

Comment

Inspiration

What it does

How we built it

Challenges we ran into### 💡 Inspiration

For over a decade, virtual reality cinema made a false promise. Flat 360-degree video isn't spatial reality—it locks the viewer's head in rotational 3DoF with zero motion parallax, causing severe disorientation and nausea. Meanwhile, commercial spatial video solutions isolate users behind flat 2D window frames behind a prohibitive $3,500 hardware wall. We asked: Why can't true volumetric cinema run natively inside the browser, completely free of bulky plastic controllers?

ChronoSplat 4D was born to redefine immersive storytelling on Meta Quest 3 through open WebXR standards, real-time 4D Gaussian Splatting, and natural human hand biomechanics.

🚀 What It Does

ChronoSplat 4D is a hands-first WebXR volumetric cinema studio running at a locked 90 FPS directly inside the Meta Quest Browser:

  • Hands-First Zero-Controller Rig: Powered by native WebXR Hand Tracking (XRHand Joints 9 & 4), fully operable from a seated cockpit position without holding controllers.
  • Air-Pinch Temporal Caliper: Pinch index and thumb in mid-air to micro-scrub through temporal volumetric keyframes with millimeter fidelity.
  • Bimanual Spatial Scale Rig: Scale dynamic 4D captures effortlessly between a 0.1x desktop diorama, 1.0x life-size realism, and an overwhelming 10x IMAX scale.
  • Contactless HRTF Audio Snapping: Point toward an actor in 3D space to isolate their directional voicebed with dynamic HRTF binaural attenuation.
  • Dual-LLM Spatial Director Copilot: Real-time conversational copilot powered by Google Gemini 1.5 Flash with automatic failover to NVIDIA Nemotron NIM for resilient, sub-second spatial camera direction.

🛠️ How We Built It

  • Frontend Core: Next.js 14 App Router, TypeScript, and Tailwind CSS with custom OLED glassmorphic components.
  • Volumetric Rendering: WebXR Device API combined with Three.js WebGL2 custom point-cloud shaders optimized for Quest 3 tile-based GPU architectures.
  • Spatial AI Pipeline: Secure Next.js server route orchestrating Google Gemini 1.5 Flash for low-latency JSON spatial intent extraction, backed by deterministic NVIDIA Nemotron failover.
  • Spatial Audio: Web Audio API with PannerNode HRTF convolution filters tied to user gaze and hand-ray vectors.
  • CI/CD: Automated builds verified via Vercel Edge Network.

🧗 Challenges We Ran Into

  • Mobile WebXR Thermal & Framerate Budgets: Rendering heavy 4D Gaussian Splatting sequences within the Quest 3 browser without thermal throttling. We solved this with dynamic splat density culling and spatial frustum budgeting.
  • Controllerless Hand Tracking Precision: Avoiding accidental gestures during seated viewing. We implemented an ergonomic 20mm air-pinch deadzone threshold with haptic visual cues.
  • Zero-Latency AI Copilot Failover: Managing API rate limits during live demos by building a resilient server-side fall-through pipeline from Gemini to NVIDIA NIM endpoints.

🏆 Accomplishments We're Proud Of

  • Eliminating controller friction completely: The entire spatial studio is operable purely via air-pinch gestures while comfortably seated.
  • Building an enterprise-grade failover pipeline between Google Gemini and NVIDIA Nemotron NIM for uninterrupted voice-to-spatial intent processing.
  • Achieving locked 90 FPS browser-based volumetric playback fidelity with zero native installation overhead.

🧠 What We Learned

  • Open WebXR standards on Meta Quest 3 are powerful enough to challenge native Unreal/Unity app-store paradigms.
  • Multimodal spatial intent parsing requires deterministic JSON schemas to reliably translate subjective film direction into 6DoF camera vectors.

🔮 What's Next for ChronoSplat 4D

  • Serverless GPU ingestion clusters (Modal / PyTorch 3DGS) for automated octree LOD chunking.
  • Direct Cloudflare R2 presigned streaming to bypass serverless payload limits.
  • Horizon Store progressive WebXR application packaging.

Accomplishments that we're proud of

What we learned

What's next for ChronoSplat 4D

Built With

Share this project:

Updates

Submission history