Immersive Moving Spaces

• 

Inspiration

Moving Spaces explores what happens when a real person enters a digital memory of a room.

The idea is to combine physical presence with reconstructed space: the performer becomes an image inside the room, and their movement changes the room itself. Instead of treating mixed reality as a way to simply place virtual objects in reality, the project explores how the environment can behave like a fragile, living memory.

What it does

Moving Spaces captures a physical room and reconstructs it as a Gaussian-splat environment inside a PICO 4 Ultra Enterprise headset.

A live performer is captured through the headset cameras without wearing a headset or motion-capture suit. Body segmentation is used to isolate the performer and place them inside the reconstructed room.

As the performer moves, their body generates soft waves and deformations in the digital environment.

The project also explores stereo depth reconstruction and multi-headset spatial capture, allowing different observations of the same room to be combined into a shared spatial representation.

The core idea is:

BODY → IMAGE → SPACE → MEMORY

How I built it

The project was built in Unity using the PICO 4 Ultra Enterprise SDK.

The main technologies include:

• Unity and URP
• PICO camera capture and spatial tracking
• MediaPipe body tracking and segmentation
• Gaussian-splat room reconstruction
• Stereo depth estimation
• GPU-based image and spatial processing
• Custom Unity shaders for masking, displacement and blending
• Point-cloud rendering
• Multi-headset spatial alignment

The system combines a low-poly spatial map for alignment with a high-detail Gaussian-splat reconstruction for the visible environment.

Challenges I ran into

The biggest challenge was latency.

The PICO camera feed has to be captured, transferred, processed, segmented and rendered before the performer can appear inside the virtual room. This makes fast movement difficult.

I also encountered unstable segmentation, stereo-depth ambiguity, textureless surfaces, lighting problems and limitations in accessing reliable depth data from the headset.

Another challenge was representing the performer naturally. Early approaches created visible strips and unwanted rectangular camera artefacts. I eventually replaced them with a continuous camera-derived actor surface controlled by a refined body mask.

Accomplishments that I'm proud of

I built a working prototype where a live performer can enter a reconstructed Gaussian-splat room and physically disturb the digital environment.

I'm particularly proud of connecting several normally separate systems:

camera capture → body segmentation → actor reconstruction → depth → spatial points → room deformation.

I also developed a multi-headset architecture capable of aligning spatial observations into a shared room coordinate system, creating a foundation for collaborative spatial performances.

What I learned

I learned that mixed reality is not only about accurately reproducing the physical world.

Small imperfections, latency and unstable reconstruction can become part of the artistic language.

I also learned how closely technical decisions affect the feeling of a performance: segmentation quality, timing, depth and spatial alignment all directly influence how believable the digital memory feels.

Most importantly, I learned to think of the camera not just as a recording device, but as a bridge between the performer and the reconstructed space.

What's next for Moving Spaces

The next step is to make the performer even more spatial.

I want to improve real-time depth and volumetric reconstruction, make the room deformation more responsive, and explore multiple performers and multiple headsets interacting within the same reconstructed space.

The long-term goal is to create a mixed-reality performance where the audience doesn't simply see a virtual room.

They see a room that remembers, reacts and changes because someone is inside it.

Built With

• pico
• unity