Music Kaleidoscope

What if you could touch the music you create and play with?

Comment

Inspiration

I have always been fascinated by the intersection of audio and spatial presence. My initial concept was a technical tool for room acoustics, but I realized that wasn't fun. The spark for "Music Kaleidoscope" came when my son, a composer, introduced me to the work of Steve Reich.

I watched a breakdown of Reich’s technique (specifically the "Piano Phase" song) which described his music as a "kaleidoscope," where complex, beautiful patterns emerge from simple, repeated objects shifting slightly out of sync. I realized that VR and Mixed Reality are the perfect mediums to visualize this. In the real world, "phasing" is an abstract concept you hear in Mixed Reality, I could let users physically walk through the phase patterns, grab the sound waves with their hands, and "conduct" the chaos into order. It transformed a theoretical musical concept into a tangible, immersive entertainment experience.

What it does

Music Kaleidoscope is an immersive MR/VR instrument that turns your physical environment into a living musical canvas.

  • Spatial Phasing: Users place "musical phrase modules" around their real-world room. As you move them, the loops drift in and out of sync, recreating Steve Reich’s famous "phasing" effect in real-time.
  • Tactile Conducting: Instead of sliders or knobs, you use your hands to physically "stretch" the time between loops or "push" a phrase to speed it up.
  • Room-Scale Composition: The app analyzes your physical space (walls, tables) and allows sounds to bounce and interleave on your actual space, creating a unique "room mix". It turns your living room into a generative symphony.

How we built it

I decided to bypass generic tools and go "100% Meta" to fully leverage the Presence Platform:

  • Audio Engine: Initially I created my own temp player an array of audioclips where some would be played earlier while others would play on a delay, however, I wasn't crazy about it and frequent audioclips produce unwanted pops on the sound too. So after that I utilized the MIDI protocol for precise rhythmic control, driving a custom soundfont engine. This allows for zero-latency tempo shifts essential for the phasing effect. Needless to say I used the Meta XR Audio SDK
  • Interaction: Built using Meta XR Interaction SDK. I mapped hand gestures (pinching, pulling) to MIDI clock data, allowing users to "scrub" through time physically.
  • Mixed Reality: I used Meta MRUK Passthrough and Scene Understanding (AI). The app uses the headset's AI-driven scene reconstruction to identify physical surfaces, allowing users to anchor musical nodes to real-world objects (tables, walls), and also use the rooms accoustic signature ensuring the "holographic" music feels grounded in reality.

Challenges we ran into

The "Commercial" Soundfont Issue: Finding high-quality MIDI soundfonts that allowed for commercial use in a hackathon project was surprisingly difficult. I had to vet multiple libraries to ensure I could legally distribute the "Entertainment Experience" I envisioned.

Visualizing the Invisible: Translating the auditory concept of "phasing" into a visual language that didn't clutter the Passthrough view was tough. I had to balance UI with the user's view of their own room.

Accomplishments that we're proud of

  1. The app uses the actual room to recreate true reverb acoustics
  2. The volume of the audio rings playing is not based on audiosource volume but on actual MIDI key velocity, a piano will should more harse when played loud and more mellow when played in a quiter manner. That contributes to a good realism on the audio field.
  3. Performs steadily in 120fps and does things to audio that most Unity apps never heard of.

The "Aha!" Moment: The first time I successfully synced two audios (first engine) and then MIDI loops and manually "phased" them apart using a hand gesture was magical. It felt like holding time in my hands actually does work and brings joy to the user! I ended up playing with that rough prototype for more than 10 minutes.

What we learned

Although I'm experienced in VR all project I got into had already been setup to use generic tools and rarely Meta tools so I though that was a good oportunity to be 100% Meta! I knew a lot of staff in theoretical level but now I got my hands dirty and really dug in

  • Meta XR Audio
  • Meta XR Interaction
  • MRUK (but still have A LOT to learn here)

What's next for Music Kaleidoscope

  • VR Mode (mostly for Quest-2 devices)
  • Save/Reset starting positions
  • Gestures
  • Virtual instruments that's big!. Imagine playing air guitar where the guitar actually slides, bends notes, does vibrato etc. I have experience ironing out VR gestures and I am confident of how great this will be ones I give it the love it deserves.
  • Height calibration for better velocity scaling
  • Spawining logic to accomodate room
  • Add more people to the team: That is an essential part to make this work. I have great set of people I worked in the past that will work for this project but were not unavailable on such sort notice (I started on this project after #Unity2025 in Barcelona where I heared about it). I need to give the app a more specific visual identity and I know just the right people to do that.
  • Virtual Piano Sequencer: A full "piano roll" editor where users can write their own Reich-style phrases.
  • Cognitive3D Analytics: Implementing analytics to understand how users move through their musical creations and what problems they face, what actions are NOT achived on their first try.
  • Market Release: Launching a free version to gather user feedback for A/B testing before a full store release at a real low price OR free + pay if you want some added feature.
  • Break auto-soundfield idea as a separate asset to be used as a plug-in by other developers.

Built With

Share this project:

Updates