Inspiration
I've always been fascinated with light shows at concerts and how the visuals elevate the experience to a whole new level. The hand interactions are inspired by DJs fiddling around with knobs and sliders, it looks so cool but I always thought it would be even cooler if they were connected to stage lights and fireworks, so I went ahead and tried to built that. Another big inspiration are my memories of watching music videos together with friends back in the 90s. I thought by giving people the option to share a creative layer on top of music videos this could lay the ground work for a new kind of shared experience.
What it does
Beat Roots XR is an interactive music visualizer that takes the audio from a songs folder or from a Youtube video playing in an integrated browser. The user adds virtual plants in their real environment that will spawn visual effects synced to the music. The user can then modify those visual effects and make them react to the music in many different ways. The sound is split into different wave bands with levels and attack events. E.g. the rotation could sync to a flowing bass level while the sparks get emitted on high attacks. Configurations can be linked to a specific Youtube video and uploaded, and anyone can see them when opening the same video and use them.
How we built it
The engine is Unity using the MetaSDK. I've mainly used agentic coding with OpenAI Codex with a parallel editor workflow so I could create effects on one while letting the agent create code and tests on the other. To optimize performance I've constantly ran comparison tests and tried out just about every shader and rendering technique I've ever used and learned new ones too. The social sharing is handled through a SupaBase database.
Challenges we ran into
The technical challenge was to keep the frame rate up while rendering passthrough, the Youtube video and still use PostProcessing and a variety of textures to elevate the visual quality.
The design challenge was which settings to expose to the user to make it easy to get started and not overwhelm them. There are literally hundreds of finetuned settings working behind the scenes, so I had to decide which ones are the most intuitive and fun to play with.
Accomplishments that we're proud of
I often made a quick build test and couldn't stop playing with it until the Quest ran out of battery. I haven't had this much fun watching music videos for a long time and got excited to always try out the current effect for a different song, experiment with it, then get excited to try that for a different song and so on. I hope I can bring that excitement to other people. I'm also proud of the visual quality. I've barely seen PostProcessing used in Mixed Reality but I think it makes everything look so much better that it's worth the effort.
What we learned
I've used this as a learning experience to see how far I can push the performance of the Quest 3 and try out all kinds of advanced rendering techniques (I've landed on a combination of GPU instancing and compute shaders). On the design front I've learned that despite the speed and skill boost of agentic coding it still takes a huge amount of iterations to get a good result. Implementing the audio frequencies took a few minutes, but getting them tuned so they are well suited for connecting them to visual effects took dozens of iterations and experiments.
What's next for Beat Roots XR
First I want to polish the core experience more, create better VFX and assets and make it as easy and intuitive to use as possible. A bit of gamification with achievements could also serve as onboarding and extra motivation. I have a lot of ideas for VFX that will be more dynamic, meaning they can interact within each other so that could add more emergent design and fun challenges and achievements for gamification. I will stay focused on making things easy, so it will never be about reaction times and difficult movements.
In early 2027 I want to publish it as a free app because having users will be the most valuable thing. For the hand interactions I want to offer interchangeable controls so I can see which ones users prefer and experiment more, basically running A/B tests. The social sharing and user generated content features will also become exponentially more valuable with every additional user.
I want to add local and online multiplayer, though the challenge here will be to communicate to the users why they can't all have their usual amount of VFX at the same time (the headset is usually at it's limit for one user and reduces effects if the frame rate drops). I'm going to try to make this an interesting and interactive part for the user for single player too, e.g. visualize the performance budget and how they are spending it right now. Once I can communicate that well I will add multiplayer.
I have a lot of hopes for the new OpenXR feature that allows for multiple apps to run at the same time, which I believe is just a matter of time until this becomes standard for any kind of headset/glasses. That would elevate my app (and their apps) a lot. I would love to integrate more music and streaming services, but even Spotify/Deezer make it pretty hard if not outright impossible to access their content in a custom app, which is why I focused on Youtube first.
Once XR becomes more common to use as a screen replacement for work, I want to add minimal modes too. E.g. place a little fountain on your desk and have it subtly react to music while working, maybe use it as a Pomodoro timer.
Lastly this app would work well for user generated content beyond just configurations. I want to make an editor that allows users to adjust the shaders and textures to create completely new effects.