Inspiration
We make Unreal Musician, a free Windows app that turns MIDI files into 3D piano performances and music visualization videos. People kept watching those falling-notes videos on a flat screen and asking to be inside them: sitting at the piano, seeing every note coming, with the pianist right next to them. VR Musician is that idea built for Meta Quest 3.
What it does
VR Musician is a seated piano experience for Meta Quest 3.
- Falling-notes highway: songs play as notes that fall toward the keys of a virtual piano, coloured per hand and track, so you always see what comes next.
- A pianist plays with you: a MetaHuman pianist performs the song with animated hands and body, right beside you.
- Your hands are the controller: hand tracking lets you press the piano keys and the floating panels directly.
- Songs: bundled pieces such as Clair de Lune, Gymnopédie No. 1, Nocturne Op. 9 No. 2 and The Entertainer, plus your own .mid files.
- Panels: a song list, a transport panel (play, stop, tempo), a track panel, and a scene list.
- Two scenes: Inside, with falling notes indoors, and Outside, with falling notes under the open sky and live weather.
How we built it
- Unreal Engine 5.8 with OpenXR and the Meta XR plugin, packaged for Android with the Vulkan mobile renderer.
- Our own MidiEngine plugin family: MIDI playback, the rhythm/falling-note grid, the piano visualizer, and pianist animation driven by Control Rig.
- The falling-note grid is drawn with instanced meshes to keep the note highway cheap enough for a standalone headset.
- The Outside scene gets its live weather from our AnyWeather plugin.
Challenges we ran into
- Desktop to standalone VR: the desktop app was built for a PC GPU. Bringing a MetaHuman pianist, a full piano and a dense note highway to Quest meant reworking how notes are drawn (instanced meshes) and adapting the scenes to the mobile renderer.
- Hand-tracked UI: panels and keys had to respond reliably to finger presses without controllers.
- Packaging for Quest: getting a signed Android release build through OpenXR, Meta XR and Vulkan settings took real iteration.
Accomplishments that we're proud of
- A playable, signed Quest 3 build of our 3D pianist and falling-notes engine, uploaded to the Meta Horizon developer dashboard.
- The same MIDI engine now drives both our Windows app and a standalone VR app.
What we learned
- Standalone VR changes how you budget every frame: instancing and fewer moving actors matter more than visual extras.
- Hand tracking makes a music app feel natural: reaching out to press a key or a panel needs no explanation.
What's next for VR Musician
-More scenes like Drummers or VJ loops to enhance music visualization.
- Continued performance work on the VR side
- Release on the Meta Horizon Store.
- More songs and scenes, and features from the desktop app.
Built With
- android
- c++
- claude
- control-rig
- hand-tracking
- meta-quest
- meta-xr-sdk
- metahuman
- midi.js
- openxr
- unreal-engine
- vulkan