Sound Orbs is a mixed reality instrument for bare hands. A window opens in your own wall. Five orbs wait behind the glass, and a pinch brings one into your room. Reach into it and your hand shapes its sound.

To try it: open the link in the Meta Quest Browser on a Quest 3 or 3S, press Start and sit facing a wall, sound on, controllers put away. If the room was scanned in Space Setup, the window sits on your real wall. The first run takes about two minutes. If you see pointer rays and no hands, update the headset's system software.

Inspiration

Most people never get to play an instrument. The first hours are wrong notes and sore fingers, and most of us stop there. At Museverse we build music learning for XR, and here we wanted the other end of that road: something that sounds good in the first minute, for a person who has never played anything.

Hand tracking gave us the way in. Our first proof, in early November, was a ball of sardines that parts around your hand the way a shoal parts around a sea lion. Reaching into it felt like touching something alive, so every sound got a body you can reach into: fish, a bubble, fire, stone and light.

We kept it calm on purpose. You sit down, nothing is scored and nothing can go wrong. It is a few quiet minutes with sound at the end of a day.

What it does

  • It starts on your wall. SOUND ORBS appears at eye level over five white rings. A ghost hand shows one pinch. You copy it, and the ring plays its note and fills with its orb.
  • It teaches one move at a time, with no tutorial screen. Pinch an orb and hold, and it comes to you. Reach in with your other hand and the sound changes with every move. A second orb comes out of the wall, and a touch on its core changes the sound. Every step waits for you and ends with a tick and a chime.
  • Then the window opens. That is the reward for the lesson. Your wall opens on a valley of cherry blossom, and from there you play freely.
  • Five orbs, five sounds, five worlds. Earth opens the cherry blossom valley, water a bait ball of sardines with dolphins, fire a bonfire on a shore at dusk, neon a city at night, and air a sky above the clouds. Stay with the earth orb and a wagtail hops out of the valley onto your own floor.

You play seated, and every move happens within arm's reach. Either hand can make every move. Once you are in, there is no menu and no button, and nothing can be failed.

How we built it

Sound Orbs is a WebXR page for the Meta Quest Browser, written in JavaScript on three.js. You open a link and nothing is installed.

  • Hands. WebXR hand input gives 25 joints per hand on every frame. A pinch starts when the thumb and index tips come within 1.6 cm and ends at 3 cm, so it does not flicker. Each hand's bones are physics shapes that touch, dent and push the orbs.
  • Room. The session is mixed reality from the first frame. Plane detection finds the walls from Space Setup, and the window goes on the nearest wall in front of you at eye height. With no usable wall it stands 2.6 m ahead. The room's measured size sets the length of the reverb.
  • Sound and worlds. The sound is synthesised live with the Web Audio API, and each orb is placed in the room with an HRTF panner that follows your head. The five worlds are real 3D scenes behind the glass, clipped to the window with a stencil mask, so they shift as you move your head.
  • Field of view. The window, the lesson and a held orb are fitted to a view 70 degrees wide and 70 high, so everything that matters stays in the middle.
  • Frame rate. The page asks the headset for 72 Hz. Fish, embers and air particles are simulated in a Web Worker, and a governor trims the heaviest effects when a frame runs long.
  • Tools. We check the lesson in Meta's Immersive Web Emulator, and the ghost hand uses the Quest hand poses from Meta's emulation runtime. Textures are CC0 or MIT-licensed, the hand model is the open WebXR generic hand, and the rest is our own code on three.js.

Challenges we ran into

  • Five live worlds on a headset. With all five in, the Quest 3 dropped frames and the sound broke up. We cut particles, moved the simulations to a worker, clipped the worlds with a stencil and made the sound engine cheaper.
  • Teaching with no words. A demonstration off to the side went unseen. A ghost hand that shows each move where you make it took a lot of tuning, and it works.
  • The orb in your hand hides the next step. A first-time player held a large glowing orb in front of her and did not see the next orb waiting in the wall behind it. Now the first orb drifts back and to the side before the second one comes out.

Accomplishments that we're proud of

  • A first-time player went from the logo to the open window in about 2 minutes 10 seconds, with nobody telling her what to do.
  • Three people took it from a first proof to this entry in 11 days and 35 builds.
  • It runs from a link, with real hands, a real wall and five worlds behind it.

What we learned

  • With nothing to touch, sound and light have to confirm every contact. Every step now ends with a tick and a chime.
  • People look at the thing in their hand. The next step has to appear there, or the thing in the hand has to get out of the way.
  • One gesture, one meaning. A quick pinch used to play an orb and a held pinch pulled it. A first-time player struggled to bring an orb to her, so now a pinch does one thing: the orb comes to you.

What's next for Sound Orbs

  • Sound Orbs goes into Museverse's school demo.
  • More sounds for each orb, and things to find in every world.
  • More first-time players and more rooms. Every test so far changed the build.
  • Target launch: June 2027 on the Meta Horizon Store.

Built With

  • audioworklet
  • css3
  • github
  • glsl
  • html5
  • immersive-web-emulator
  • javascript
  • meta-quest-3
  • meta-quest-browser
  • passthrough
  • poly-haven
  • sharedarraybuffer
  • spatial-audio
  • three.js
  • vercel
  • web-audio-api
  • web-workers
  • webgl
  • webxr
  • webxr-hand-input
  • webxr-input-profiles
  • webxr-plane-detection
Share this project:

Updates

Submission history