Inspiration

A stem player gives you four faders: vocals, drums, bass and everything else. It's one of the most fun ways to take a song apart, but it's still a flat box with sliders. I wanted to know what it becomes when your hands are the faders and the room around you is the mixer. In mixed reality, a part of a song doesn't have to be a slider. It can be a thing you hold, put down somewhere, and hear from exactly where you left it.

What it does

Unmix lets you pull a song apart with your bare hands and remix it by placing its parts in the room around you.

  1. Pick a song. Poke a record on the shelf. The song appears in front of you as a glowing sphere and starts playing.
  2. Pull it apart. Grab the sphere with both hands and pull. It bursts into four orbs (for example drums, bass, lead and guitar), and the music never stops. A Split button does the same thing.
  3. Place each part. Every orb is a spatial sound source. Where it sits is where you hear it, and closer is louder. Mixing becomes arranging the room.
  4. Shape it. Poke an orb to open its controls: mute, solo, darker, brighter, pitch and echo. Every gesture has a large poke-button fallback.
  5. Ask the producer. Three suggestion chips propose moves written for that song, such as "Bass up close, drums wide." Poke one and the orbs glide into place in front of you, with a one-line reason and Keep / Undo.
  6. Add yourself. Record two bars from the microphone, in time with the song, and they become a fifth orb.
  7. Mix it down. Push two orbs together, or poke Mix down, and the parts fuse back into one sphere. Your mix is saved for that song.

Everything is hands only, seated, in passthrough, and stays within arm's reach in front of you.

How I built it

  • Meta Immersive Web SDK (IWSDK) with three.js and its built-in grab, pinch and poke interactions. No custom gesture recognition, so it's robust when tracking misreads a pinch.
  • Web Audio API. Each part runs through a low-pass filter, a tempo-synced echo and a gain stage into an HRTF PannerNode that follows its orb every frame. All parts start on the same audio clock, so they stay sample-locked.
  • One mix-state object is the single source of truth. Hands and the producer write to it; audio and visuals read from it. A saved mix is a snapshot of it.
  • Producer actions go through a strict, validated action list (move, volume, mute, solo, filter, pitch, echo, half-time, loop section). Nothing outside that list can run, and every change can be undone.
  • AudioWorklet loop recorder that captures sample-accurately on the bar grid and corrects for audio latency.
  • Demucs splits each song into parts offline. Songs are by Kevin MacLeod (incompetech.com), CC BY 4.0.
  • Hosted as a static site on GitHub Pages: nothing to install, open it in the Meta Quest Browser.

Challenges I ran into

  • No headset. Everything was built and tested in the IWSDK emulator, so I kept to the SDK's built-in interactions and gave every gesture a poke-button fallback.
  • Keeping four parts in sync while each can be filtered, echoed, pitched or looped on its own.
  • Instrumental songs have no vocals. I used Demucs's six-part split and let each song name its own four orbs, so no orb is ever silent.

Accomplishments that I'm proud of

  • The pull-apart moment: one gesture, and a song becomes something you can walk your hands through.
  • Spatial mixing that you hear, not just see.
  • Producer suggestions that act in front of you and are always undoable.

What I learned

Spatial audio does a lot of the interface work. When sound comes from where an object sits, people understand the mix without labels or meters.

What's next for Unmix

  • Live AI producer suggestions from a model, through a small relay that validates every action.
  • Pinch-and-drag shaping gestures alongside the buttons.
  • Bring your own song, and MIDI controllers for hands-on knobs.

Try it

Live: https://amongstwolves.github.io/unmix/ (open in the Meta Quest Browser and press Enter XR; turn on hand tracking) Code: https://github.com/amongstwolves/unmix Demo video: https://youtu.be/Eh4X2sVBGE0 Music: Funkorama, Hyperfun and Groove Grove by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0. Demo video recorded in the IWSDK emulator.

Built With

Share this project:

Updates

Submission history