Stencil VR

Inspiration

Stencil VR has grown into an essential tool for a massive community of professional muralists and artists. As we listened to their feedback, two requests stood out above the rest: the creative need to break free from flat surfaces to trace onto curved objects, and the technical need for absolute stability against drift. This update was inspired by our commitment to those artists, we wanted to eliminate the technical frustrations of XR tracking and unlock new geometric possibilities, allowing them to create on any surface with total confidence.

What it does

Stencil VR transforms the Meta Quest into a precision art projector. This update unlocks Curved Stencils, allowing artists to wrap digital reference imagery around physical columns and objects. We achieved "Zero Drift" stability via a new Multi-Anchor System, a Precision Trace Camera that eliminates passthrough distortion, and advanced Depth Calibration tools to eliminate parallax. We also solved the immersion-breaking issue of hand visibility: artists can now use Hand Occlusion to see their physical hand and pen overlaying the virtual stencil without clipping errors.

How we built it

We rebuilt the anchoring architecture to use an array of spatial anchors that prevent single anchor drift from misplacing the stencil, which is especially important for the massive murals our users often work on. For the Precision Trace Camera, we fused raw feeds from both Quest cameras to eliminate reprojection distortion. We implemented a depth map that extensively utilizes the environment raycasting feature to get accurate depth measurements for each part of the stencil. Hand occlusion was achieved by implementing a reverse-mask filter within a tracked spherical mesh around the user's hand, isolating depth calculations to avoid clipping the wall behind the stencil.

Challenges we ran into

Hand Occlusion presented a technical Catch-22. Due to depth sensor noise, if we placed the stencil flush with the wall, the physical wall would incorrectly occlude the artwork. If we moved the stencil forward to prevent that, the user's hand and pen would render behind the stencil while tracing. Since raw hand tracking wasn't precise enough for a clean cutout, we engineered a hybrid solution: combining depth occlusion with a hand-tracking mask to isolate the effect. This also required building a custom Depth Calibration system to account for hardware variance.

Accomplishments that we're proud of

We are incredibly proud of Hand Occlusion; solving the Z-fighting between the wall and the user's hand required a complex mix of masking and depth filtering that feels like magic to use. We are equally proud of the Precision Trace Camera, which bypasses standard rendering to deliver a raw, distortion-free view of the world. Additionally, the Depth Display Map stands out for successfully turning abstract sensor noise into an intuitive visual guide, not only empowering users to solve parallax issues themselves but also enabling easy rotational and curve alignment on any surface.

What we learned

We learned that hardware isn't uniform. Depth sensors vary between headsets and lighting conditions. This taught us that we can't rely on auto-detection alone; building the Depth Calibration Flow and manual offsets was essential to account for hardware variance and ensure professional-grade accuracy for every user, regardless of their specific device or environment.

What's next for Stencil VR

We plan to expand our 3D capabilities beyond simple curves to fully complex geometries and custom models, opening up the use cases for Stencil even further. We are also exploring colocation and shared anchors, allowing multiple artists to work on the same stencil simultaneously. Our goal is to continuously improve Stencil VR, empowering artists to focus less on the technology and more on their art.