Inspiration
Visual programming tools are usually confined to two-dimensional screens. I wanted to explore what would happen if I created one for a three-dimensional space. Inspired by shader programming, Minecraft Redstone and creative coding tools such as cables.gl, I started building Spatial Nodes, a visual programming language designed specifically for virtual reality.
What it does
Spatial Nodes is a visual programming environment for creating interactive 3D experiences directly in VR.
You can grab nodes from a catalogue, place them in space, and connect them using virtual cables. Each node performs a small operation, such as adding vectors, calculating trigonometric functions, accessing hand positions, or drawing 3D geometry.
Node connections define how values flow through the scene. Since everything is evaluated continuously, you can create animations and interactive experiences without explicitly programming events or triggers.
How I built it
I used LÖVR, a framework for creating VR experiences with Lua. The nodes are compiled to WASM and executed using Wasmtime. The application also uses Meta's spatial APIs to access information about the physical environment.
Challenges I ran into
LÖVR doesn't support using Meta's Scene API yet, so I had to create my own plugin that uses OpenXR APIs. I've never done this before and it took quite some time to figure out.
Another challenge was performance. I first evaluated the node trees in Lua which was too slow for more complex scenes. Moving to WebAssembly fixed that, but it took a lot of effort and there's still room for improvement, such as instanced rendering.
Accomplishments that I'm proud of
I'm proud of how the project has evolved from an initial experiment into a working creative coding environment. Before this challenge, my project was a proof of concept. There were only a small selection of nodes and there was no UI for adding nodes without the headset being connected to a computer.
I'm very happy with various UX improvements I made (output previews, warnings, multiple node outputs) and the new nodes i've implemented. I especially like the new nodes that let you access the individual finger joints and the nodes that let you access the surfaces of your play area (like room walls, desks etc.)
What I learned
This project taught me a lot about VR interaction design and compiler development.
What's next for Spatial Nodes – Creative Coding in XR
My next priority will be introducing reusable building blocks such as node groups or functions. Currently, complex node trees can become very messy. I want most of the core nodes to be implemented as node groups, except for foundational operations such as scalar math.
At some point I'd like to introduce interactive tutorials, more example scenes, and guided challenges that teach programming concepts through experimentation. I also want to introduce social features (sharing scenes, exploring scenes from other users) at some point.
I hope to publicly launch a stable-ish version in the second half 2027. I think Spatial Nodes could become a tool that makes coding in VR/XR expressive, creative and fun.
