Inspiration

I'm a huge fan of attending shows at spatial audio venues. The music showcased in these venues can be extraordinarily dynamic and responsive, but any chance for audiences to move around the space goes out the window the second a performing artist stands in place with a projector behind them. As for the artists, their music cannot change spatially in real time unless a dedicated audio engineer controls visualized audio objects on a flatscreen monitor. As someone who loves to mix spatial audio, I wanted spatial audio engineers and producers the ability to visualize 3D audio objects and remix them in their digital audio workstation. MRig, short for both "Mixed Rig" and "Music Reality, I guess", aims to break both of these barriers down by making the musician their own audio engineer while giving them the freedom to move around a 360 degree space to interact with their audiences.

What it does

MRig brings a musician's entire performance rig into their headset via passthrough. Users can switch between loops of their songs in real time, adjust volume of each track, solo/mute/link loops, route audio to fit different channel configurations, and move audio objects around a room, all with hand tracking, controllers, and/or eye gaze. The software can accommodate accessibility needs for multiple users, even musicians using instruments that require two hands by utilizing eye gaze control. Users can configure the UX to match their level of comfort by mapping gestures to different actions. A feature that any musician sharing a bill with others will appreciate is the inclusion of a setlist timer that can display how much time the musician has left in their set, displayed in their eyes or in their environment. If a producer likes the layout of a current scene, they can save it in an anchor menu to return to it later. Musicians can choose whether to use occlusion to see their real hands and instruments over virtual objects or draw meshes over their hands depending on their preferences. One of the latest developments has been mapping the controls of MRig into REAPER, a digital audio workstation (DAW) that takes OSC messages from the headset and lets me remix spatial audio with the benefit of visualizing the audio objects in real time. This part is still a work in progress but finished enough to showcase for this competition.

How I built it

I am a solo developer who built this in Unity with Wwise as the audio middleware. I first composed original music, recorded it in Cubase, exported the stems into Wwise, then mapped certain playlists to various states in the song to optimize song section switching. I then tied each instrument to RTPC parameters for volume for seamless remixing, then made one shot sound effects that fit within the key of the song. Via the Tempest 3D audio plug-in from Wwise, all of the music stems and one shots can render in spatial audio. In Unity, I used assets purchased from the Asset Store to create low-poly visuals of the instruments as well as implement faders and solo/mute buttons. I mapped each instrument and fader to the appropriate instrument in Wwise, then simple colored cubes to serve as triggers that switch between a verse, chorus, and interlude on a whim. Using ray casting and hand tracking, I implemented interaction systems to accommodate a variety of users rather than forcing them to use my most comfortable hand gestures.

Challenges I ran into

Approximating user comfort when utilizing an instrument has been an ongoing challenge with the development of this project. Originally, I did not draw any meshes over the user's hands, but then my guitar and arms would vanish behind virtual objects when I was playing! To fix this, I implemented occlusion via Unity's OpenXR occlusion rendering component, making it possible for me to see both my hands and my guitar when using the software even when overlaid on top of virtual objects. Another problem was that, as I would play my guitar, the headset would recognize the distance between my left index finger and thumb as a menu button input, planting my palm menu directly over my fretboard mid solo! To fix this, I implemented 2 different menu types when the user presses the menu button: float spawns a moveable and scalable menu by the user's palm, and forearm mode (my preferred method) keeps the menu on the back of my forearm constantly, meaning I only need to use it when I'm done playing the guitar.

Accomplishments that I'm proud of

Even though this was a simple plug-in to implement, I am most proud of spatializing the audio of this tool. Considering that the goal of this project is to create new types of engagement between artists and audiences, the fact that this music can be spatially remixed in real-time gives me hope for the future potential of interactive live music performances. Even when I'm not actively playing guitar in the scene, moving my audio stems around me beats remixing a binaural rendering on headphones any day. It really sounds like my drums and bass are behind my head when I turn my back to them, and throwing one-shot instruments around the room makes me excited for spatial performers to utilize this tool. I'm especially proud of the recent developments with DAW mode, letting me pair my headset to REAPER and control an audio session via OSC messages.

What I learned

While the XR music ecosystem is limited at this time, its infancy opens up many doors for how music can be created going forward. Past winners like Inkphony and (m)ORPH Mixed Reality each use mixed reality and spatial computing in drastically different ways for completely unique experiences, but they each offer a type of music experience that was impossible prior to their development. Sharing and testing this app with friends, family, and now the Meta VR Start Developers competition has done so much to evolve this project from an incomprehensible idea I formed 3 years ago, before I ever even heard the term "mixed reality," into a genuine tool for performers. I hope MRig offers musicians a new way to bring their creative visions to life and for audiences to partake in new interactions with their favorite artists.

What's next for MRig

I have three definitive goals in mind for what comes next in this project's development, the first of which will be multiplayer functionality. For this project to truly go next level, I want song sessions to be open to any users in a room with HMDs, potentially changing audience dynamics from active listeners to directly contributing to the music-making process with musicians! Secondly, I want to give artists a platform to upload their own songs into the software rather than using solely the music I put into the app myself, likely by further developing DAW integrations to other softwares like Ableton Live and Nuendo, and creating a separate companion desktop app to upload their stems and one-shots to their own account: modifying these stems should be done either in the desktop app or the headset directly. Finally, I am studying Extended Realities with a concentration in Wireless Networking at Northeastern University at this time, all with the goal of learning how to connect spatial audio systems to HMDs and other XR tools. I want to explore how to reduce audio latency across multiple XR devices for greater musical collaboration. I'm excited to see how MRig and other XR music tools shape the future of music experiences!

Built With

Share this project:

Updates

Submission history