Inspiration
Every new way of watching movies has pulled us deeper into the story. The cinema gave us the big screen, IMAX filled our view, and 3D and surround sound added depth and space. But in the end you're always looking at the film from the outside.
VR should be the next step, and it can go further than any of them. Yet today's VR movie apps give you a flat rectangle floating in the same generic lounge, scene after scene, film after film. Even long-time XR users stopped putting the headset on for it.
So we asked: what if you weren't watching the movie, but standing inside it? When the film cuts to a city at dawn, that city wraps around you. When the hero walks into a moonlit forest, the trees are right beside you. No cinema, no IMAX and no TV can do that. That's the experience we set out to build.
While custom 3D environments are possible, creating these for every scene would be close to impossible in scope. We wanted to create a plug-and-play system, where any movie (or even your own images) could be transformed into beautiful environments.
What it does
Immersed Stories is a hands-first spatial cinema for Meta Quest. As the story moves, the world around your screen moves with it:
- 🌍 A new world for every scene. AI-generated 360° environments match each scene's setting, lighting, mood and art style. An animated film gets an animated world.
- 🧊 Real 3D, not a flat skybox. Every world comes with its own depth map, processed locally on the headset. Trees, walls and mountains have true 3D depth.
- 🎬 Cuts become transitions. When the movie changes location, the colours and the 3D shape of the world blend smoothly into the next one.
- ✋ Touch the story. Touch nearby scenery with a fingertip, or pinch at a distant mountain. The world ripples and gives off glowing particles in the colours of the scene. This gives you a feeling of being inside the virtual world.
- 📂 Bring your own movies and images, including stereoscopic 3D (side-by-side and top-bottom). Our server generates worlds for your own videos while you watch. To keep queue times down, limited to 4 skyboxes per video, and one per image.
- 🛋️ Made for the couch and the airplane seat. Grab the screen to move it, drag its corners to go from TV to IMAX size, and turn the 3D depth up or down from a wrist menu. No controllers needed.
How we built it
- Our own open-source AI pipeline (SkyboxSystem). It uses only open-weight models in Docker on our own NAS GPU, with no paid AI services:
- Qwen3-VL reads the frame and its visual style.
- FLUX.1 with the DiT360 panorama LoRA paints a seamless 360° world.
- Mask R-CNN and OWLv2 detect people and creatures, and FLUX paints them out, so the world is an empty stage instead of copies of the actors.
- SegFormer finds the sky, and MoGe-2 estimates depth from 12 views merged into one panorama.
- Real-ESRGAN upscales the result to 4096×2048.
- Two quality levels: fast (about 2.5 minutes) for your own videos, and quality (extra detail passes) for our pre-made film library, loaded with Addressables.
- Unity 6.7 beta: we chose the newest engine release for its high-end rendering features and Quest optimizations. That's what lets a 4K movie and a fully 3D world run side by side on a standalone headset.
- OpenXR (Meta) with XR Hands and XR Interaction Toolkit for hand tracking, poke, pinch, grab and the wrist menu.
- AVPro Video v3 for smooth 4K and stereo 3D video playback on Quest.
- On the headset: a custom depth shader on a 335k-vertex sphere. Floor detection and frame scoring run in Burst jobs with zero garbage collection. The scoring picks which frames of your video are worth turning into a world.
- Claude as our AI pair programmer throughout.
Challenges we ran into
- Quest can't run diffusion models, and cloud AI is slow or expensive. We built our own queued GPU server and fit the whole pipeline into 12 GB of VRAM by loading one model at a time.
- Making panoramas feel 3D. In generated worlds the camera can sit anywhere from 0.6 m to 36 m above the ground. Merging depth from many views and measuring the ground on both the server and the headset makes every world feel like you're standing in it.
- Keeping the movie the hero. The world must always draw behind the screen and your hands, but still hide its own far parts correctly. We solved this in the shader by squeezing the world's depth range.
- Minutes of generation against seconds of scene. Our film library comes with worlds made in advance. Your own videos get worlds generated while you watch.
Accomplishments that we're proud of
The wow effect is back. People who have used XR for years react like it's their first time in a headset when the room becomes the film's world. Then they reach out to touch it. The whole AI stack behind it is open-source, open-weight and self-hosted. Getting the whole AI stack to run locally, using only open-weight AI models was also a very big milestone to achieve, We hope this workflow can be used by other in their applications as well, to create more unique experiences without relying on big commercial platforms.
What we learned
Self-hosting a multi-model AI pipeline in Docker on consumer hardware was the buggest hurdle to overcome. Shaping 3D worlds from depth maps in a vertex shader was a fun challenge to get more advanced vertex shaders working. Getting the most out of the newest Unity release on Quest was fun and hopefully improved Unity with over 20 bug reports. Designing hands-first, seated interactions that feel natural. It was great to see people pick up the headset and smile within seconds
What's next for Immersed Stories
Faster generation for personal videos, aiming for near real time. A growing library made with filmmakers. Watch parties where friends share the same world. A subscription model for hosted generation, which should improve quality and speed for your own content.

