TensorView VR — Explore Neural Networks in Virtual Reality
Inspiration
Neural networks are becoming increasingly complex, but developers still rely heavily on traditional 2D diagrams and technical tables to understand their architecture.
I wanted to explore a different approach: what if developers could step inside a neural network and inspect its computation graph as a spatial environment?
That idea became TensorView VR: an attempt to make neural network architecture easier to explore, understand, and analyze through virtual reality.
What it does
TensorView VR is a spatial ONNX graph explorer and static compute analyzer designed for Meta Quest.
It transforms ONNX computation graphs into interactive 3D structures. Each layer becomes a spatial block connected to other operations through visual data-flow links.
Developers can select individual layers and inspect:
- Input and output tensor shapes.
- Parameter counts.
- Estimated floating-point operations (FLOPs).
- Estimated FP32 activation memory.
- Computational hotspots across the model.
The application includes three synthetic demonstration architectures: a ResNet-style CNN, a MobileNet-style network, and a Tiny Transformer. Users can also load their own ONNX models.
How I built it
The project combines a JavaScript WebXR application built with Three.js and IWSDK with an experimental native C++17 implementation using OpenXR and OpenGL ES 3.
The WebXR application includes:
- Spatial 3D graph visualization.
- Hand-oriented interaction.
- An interactive layer inspector.
- Computational heat maps.
- High-contrast accessibility mode.
- Guided onboarding.
- Procedural sound effects.
A dependency-free ONNX protobuf parser is implemented in JavaScript, alongside a corresponding native C++ parser.
I also created demonstration models to test graph parsing, layout, and computational estimates.
Hand interaction
Hand interaction is central to the intended VR experience.
Users can point at graph elements using Quest's far-field hand pointer and pinch to select individual layers.
Pinching empty space allows users to grab and move the graph, while two-hand interaction supports zooming.
Controller input is also supported as a fallback.
Challenges
One of the main challenges was transforming a conventional computation graph into a readable spatial structure.
Neural networks contain branching paths, residual connections, and many different operations. The visualization needed to communicate these relationships without becoming confusing.
Another challenge was implementing ONNX parsing without relying on a large external runtime, while keeping the JavaScript and C++ implementations consistent.
Designing interactions around hand gestures also required thinking beyond conventional mouse and keyboard interfaces.
What I learned
This project helped me explore the intersection of machine learning tooling, spatial computing, and VR interaction design.
I learned more about ONNX graph structures, protobuf parsing, computational cost estimation, WebXR development, OpenXR architecture, and designing interfaces for spatial environments.
It also reinforced the importance of making technical information understandable, rather than simply displaying more metrics.
What's next
Future development will focus on validating the experience on physical Quest hardware, improving graph navigation and interaction, expanding ONNX operator coverage, and adding more detailed model analysis.
I also want to investigate runtime performance metrics and live inference visualization, going beyond the current static estimates.
My long-term goal is to make TensorView VR a practical spatial debugging and model-inspection tool for machine learning developers.
Built With
- c++17
- computing
- es
- hand
- iwsdk
- javascript
- learning
- machine
- meta
- networks
- neural
- onnx
- opengl
- openxr
- quest
- reality
- spatial
- three.js
- virtual
- vite
- webxr