OpenQuestCapture

Project Story

Inspiration

When Meta launched Horizon Hyperscape, I was impressed by what Quest 3's spatial sensors could achieve. As someone who's spent years building 3D tools at Pixar and Microsoft, I saw incredible potential: Quest 3 has professional-grade depth cameras, RGB capture, and precise tracking - all in a $500 device that costs 1/10th what professional spatial capture sensors cost.

I wanted to explore what else could be built with Quest's spatial capabilities. What if developers, researchers, and creators could access this raw data directly? What new applications could emerge if Quest's spatial computing power was available through open, standard formats?

OpenQuestCapture is my answer: unlocking Quest 3's spatial sensors for the wider developer community.

What It Does

OpenQuestCapture is an MIT-licensed Quest 3 app that captures raw spatial data: depth maps, RGB images, and 6DOF pose information. While capturing, users see a live 3D point cloud visualization showing exactly what areas they've covered and from which angles - solving the common problem of incomplete captures.

The raw data exports to industry-standard COLMAP format, making it compatible with Gaussian Splatting pipelines, photogrammetry software, and custom 3D reconstruction tools. Users can process everything locally or upload captures to vid2scene.com/upload/quest for cloud processing. This opens Quest 3 for use cases beyond consumer applications: architectural documentation, academic research, industrial site mapping, and developer experimentation.

How I Built It

I built OpenQuestCapture using Unity and Quest's Passthrough API to access the depth and RGB camera feeds. The core challenge was synchronizing depth, color, and pose data at capture time while maintaining smooth frame rates for the live visualization.

The visualization system renders captured points in real-time using Unity's particle system, color-coded by capture angle to help users identify gaps in coverage. I implemented a custom point cloud export format that converts to COLMAP's camera and point format for downstream processing.

Challenges

The biggest challenge was file size - raw captures generated gigabytes of data. I'm implementing QOI compression for RGB data (50%+ reduction) and exploring depth map compression techniques that preserve spatial accuracy.

The second challenge was creating intuitive capture UX. How do you show users what they've missed without cluttering the view? I'm still iterating on this - the current visualization works but needs refinement.

What I Learned

Building for Quest taught me the importance of balancing quality with performance. Every feature decision impacts frame rate, which is critical for comfortable VR. I also gained deep appreciation for how powerful Quest's spatial computing hardware is - Meta has built something truly remarkable at an accessible price point.

What's Next

Implement advanced compression (QOI for RGB, custom depth encoding)
Extract and export the high-quality triangulated mesh generated during reconstruction
Improve capture visualization to better show coverage gaps
Build processing presets for common use cases (architectural scanning, object capture, outdoor environments)
Explore integration with Meta's upcoming spatial computing APIs