ReVision

Transform any 2D image or 3D Gaussian splat into an immersive world you can explore and restyle in real time across VR and WebXR

Comment

Inspiration

A photo freezes a moment, but it can't hold what that moment felt like. You can look at a picture of your childhood home, a landmark you visited as a kid, or a place that no longer exists but what if you could revisit it?

That gap between looking at a memory and the feeling of being there is where ReVision started. We wanted to give people a way to step back into a photograph, explore it, and see it in a new light. We also were inspired by the potential for how reimagining a stimulated 3D space could allow an artist to see their own work reimagined instantly or an exhibit planner to imagine a space with a different color.

What it does

ReVision turns any 2D image or gaussian splat into an explorable, restylable immersive world across VR and WebXR.

  • Real-Time Spatial Stylizer: Upload a photo, extract its depth, reconstruct it as a gaussian splat, and step inside it in XR.
  • Creative Reimagining: Walk through a personal photo as a fully realized 3D space, then reimagine that location in a different visual style.
  • See Real Landmarks in a Different Lens: Load an existing gaussian splat of a real place and preview it under new lighting, seasons, or artistic styles while keeping the original composition intact, useful for planning, pitching, or just dreaming.
  • Immersive Creative Exploration: Artists can drop their own artwork in, convert it into a splat, and reinterpret it with new colors, textures, and styles in real time, exploring creative directions without ever starting from scratch.

In our prototype, you fetch a photo album, choose a world that's already been generated from it, and step inside using a spatial VR interface with floating style "cubes" and an extended spatial UI menu let you preview and apply new aesthetics to the world around you instantly.

How we built it

We built the experience in Unity 6 with URP around three layers: AI-generated worlds, a real-time stylization engine, and a VR game loop. Each room is a Gaussian splat environment generated from text or image prompts using WorldLabs and rendered through its Unity OpenXR splatting package. A central hub connects the rooms through portals, while a lightweight RoomManager keeps one room active at a time and handles fades, teleportation, and transitions.

The stylization engine combines a fullscreen URP ScriptableRendererFeature post-process with a custom HLSL splat shader. It supports comic halftones, RGB-offset ghosting, histogram-driven ink shading, palette snapping, and color grading. An on-GPU analyzer processes splat positions and colors using k-means clustering in Oklab perceptual color space, luminance histograms, and object IDs. These results drive the Ben-Day dots, smart ink, and palette effects, transforming photorealistic scenes into coherent pop-art environments instead of applying a simple visual filter.

The game loop centers on picking up objects to transfer their styles to the current room. Glowing keys apply contrasting looks, such as neon comic or noir, and swapping keys changes the environment instantly. The same scene supports desktop and VR controls through a PlayerBootstrap system, OpenXR tracking, controller interaction, and an in-headset styling interface. We targeted PICO and Meta Quest with ARM64, IL2CPP, and Vulkan APK builds, while testing on PICO hardware and the Meta XR Simulator. To accelerate development, we also created a Node.js Unity-scene MCP server that can inspect scenes, read console output, capture screenshots, align environments, and start builds programmatically, with the complete shared project versioned on GitHub.

Challenges we ran into

Gaussian splats contain millions of unordered colored points without objects, materials, surfaces, or semantic regions, which makes meaningful styling and collision difficult. We built a GPU-backed analyzer that clusters splats into perceptual color groups using k-means in Oklab space. Because Unity clears [NonSerialized] group data in Play mode and builds, we detect that loss and deterministically rerun the analysis at runtime. We also created tools that read each splat’s bounds and align invisible floor colliders and spawn points with the generated environment.

Rendering splats reliably in VR introduced several platform-specific challenges. The renderer requires URP Compatibility Mode, which Unity 6 controls through the URP_COMPATIBILITY_MODE scripting define. Without it, Android rooms silently render empty. Supporting both PICO and Meta Quest through OpenXR also required per-build configuration so each platform enables only its correct runtime loader and controller profiles, avoiding conflicts between Meta, PICO, and Oculus Touch features.

PICO emulator testing exposed a deeper architecture limitation. Its OpenXR runtime is ARM64-only while the emulator is x86_64, forcing XR builds through ARM translation. We verified that the ARM64 build initializes a genuine immersive OpenXR session, but per-frame splat sorting through the emulated ANGLE-to-Vulkan graphics stack caused device resets and frame starvation, worsened by injected Vulkan overlays. The full pipeline works end to end, but full-fidelity performance requires physical hardware rather than an emulated GPU.

Accomplishments that we're proud of

  • Achieving a new real-time Gaussian splat stylization engine built from scratch
  • A variety of stylizations including: comic halftones with selectable dot shapes, RGB-offset ghosting, luminance-driven ink, palette snapping, and full color grading
  • An on-GPU analyzer makes these effects possible by converting unstructured splat data into perceptual color groups, luminance histograms, and object regions

What we learned

Working with Gaussian splats taught us how different they are from meshes. They are visually rich but contain no geometry, materials, or semantic structure, so features such as collision, selection, and stylization require reconstructing information that is not inherently present. Running k-means clustering in Oklab perceptual color space was a key insight because it transformed photorealistic captures into coherent, art-directable palettes rather than visual noise. We also learned how to build custom URP renderer features and HLSL post-processing passes, configure Render Graph compatibility, and order effects so the final image resembles a printed comic instead of a stack of filters.

On the platform side, we gained a deeper understanding of OpenXR runtime discovery, VR application requirements, and architecture-specific native libraries. An ARM64 versus x86_64 mismatch can determine whether an application launches as an immersive experience or a flat window. We also discovered how desktop overlays can interfere with Vulkan and how demanding Gaussian splat rendering is for emulated GPUs. Finally, building a Unity-scene MCP bridge allowed us to inspect scenes, capture screenshots, and control the editor programmatically, turning slow manual iteration into repeatable and verifiable workflows.

What's next for ReVision

We want to build out the full pipeline we scoped but didn't have time to prototype: a WebXR website where anyone can upload their own photos, an API that generates the splat automatically, and a plugin layer so any game or app can fetch a generated world and restyle it on demand.

We also want to expand ReVision beyond personal memories toward architects and urban planners previewing a space in different lighting or seasons, museums letting visitors reimagine an exhibit, and artists using it as a genuine creative tool for rapid style exploration. Ultimately, we want ReVision to be the layer that sits between any photo and the world it's hiding inside.

Built With

Share this project:

Updates