Inspiration
Practising alone is quiet. A metronome keeps time but never listens: it doesn't get louder when you dig in, doesn't fill when your phrase turns, and doesn't stop when you do. Hardware "smart band" pedals exist, but they sit on the floor and you talk to them with your feet. In a headset the drummer can actually be there: on your rug, beside your piano, looking back at you. Musicians cue each other with their heads because their hands are busy, so that's how you talk to this drummer too.
What it does
- Sets up in your room. On entry the drummer appears at once beside your seat. Using the headset's scene understanding (floor, walls, furniture) it moves to the first clear spot and stands on your real floor, on a rug with its own shadow. A persistent anchor brings it back to the same place next time. Point and pinch to move it.
- Listens through the headset microphone. Piano, guitar, bass, voice, horn, or a song playing on a speaker. It finds the beat, the bar line and the phrase as you play, with no buttons and no settings.
- Count it in with four nods. Look at the drummer and nod four times. The sticks click on your nods, the drummer takes over the count, and comes in on the one at your tempo.
- Cue it with your head. Flick up-right for a drum break, up-left to bring it down, chin up to build, flick right for half-time, flick left to end, shake to stop. Cues land at the end of the phrase, like a real drummer. A light on the kick drum shows when a cue is pending and when it is carried out.
- Plays like a drummer. The grooves are bars of recorded human drumming, chosen to match your tempo, feel and energy. It fills at phrase ends, crashes into new sections, and goes to brushes or cross-stick when you play softly.
- Stage mode. Your room becomes a venue with lights on the beat and a crowd. The floor around your seat stays your real room, so you still see your instrument and your hands.
- Hands only, seated, no controllers. Every head cue also has a button, and the count-in can be tapped, for anyone who can't or doesn't want to nod.
- Nothing with you? A demo player (pianist, guitarist, or singer with a synthesised voice) sits where you would and plays an original song. The drummer hears it through the same listening path as a microphone, so you can see how it follows.
How we built it
- IWSDK (WebXR) on Meta Quest 3/3S, passthrough AR, hand tracking. Plane and mesh detection drive placement. A persistent room anchor holds the set in place. A UIKitML console with Lucide icons works by hand ray and pinch. Head pose runs at the display rate into a cue detector that predicts each nod's bottom, so the count-in click lands on the nod.
- A listening engine written for the browser. An AudioWorklet does feature extraction: onset strength, a 64-band mel spectrum and pitch classes. A small recurrent beat network (~270k weights, hand-written TypeScript inference) feeds a forward-filtered beat tracker and a bar-cycle tracker. A per-bar song-form network reads phrase position and sections ahead of time.
- Hearing itself, solved. The Quest's speakers sit next to its microphone, so the drummer hears its own kit louder than your music. Left alone, it would lock onto its own playing. The band's output is the reference for an echo remover in the worklet. It finds the speaker-to-mic delay by correlation, then subtracts the band magnitude-wise with per-bin adaptive filters. On top of that, the tracker ignores what it hears in the instant each of its own hits arrives. On our bench (24 songs, the drummer's own sound 10 dB louder than the music in the mic), beat accuracy after the count-in went from 25 to 62. With no self-hearing it is 81, and the remover costs nothing there.
- Measured, not guessed. Every listening change was scored offline, with no look-ahead, against annotated recordings: GuitarSet, Groove MIDI, POP909, AAM, Slakh, Jamendo/FMA Creative Commons songs.
- Drummer. Grooves come from the Groove MIDI dataset (CC BY 4.0), chosen by tempo, energy and complexity. Fills, crashes and size changes follow the phrase and energy readings. Instrument samples are rendered from the MuseScore General soundfont (MIT).
- Ghost players. The kit is procedurally modelled. Ghost hands grip the sticks, and every stroke lands on the audio clock, corrected for output latency.
Challenges
- A drummer that hears itself. We built a test bench that feeds the drummer's own hits back through a small-speaker model, and designed the echo remover and the self-hearing gate against it.
- Knowing where "one" is. Counting in with nods fixes the start, but the drummer still has to keep the bar and phrase as the music moves. The song-form network and the phrase counter carry that.
- Developing without a headset. Everything was verified in the IWSDK Quest 3 emulator with hand tracking: an automated walk-through runs the whole flow, from landing to count-in, cues, stage mode, leaving, and coming back to a restored room anchor.
What's next
- Test and tune on Quest 3 and 3S in real rooms: microphone processing, speaker-to-mic latency, nod detection on real head tracking.
- Optional USB-MIDI input for digital pianos (exact notes, no microphone).
- A bassist and a keys player, following the same listening, for full-band practice.
- Practice tools: save a jam, loop a section, slow-down practice with the drummer.
- Retrain the listening models on commercially licensed data only, ready for a paid release.
Target launch: Meta Horizon Store (as a packaged web app) in spring 2027, alongside Meta's new headsets.
Built With
- and
- cloudflare-pages.-data-and-assets:-groove-midi-dataset-(cc-by-4.0)
- commons
- creative
- fma
- for
- from
- guitarset-(cc-by-4.0)
- iwsdk
- jamendo
- musescore-general-soundfont-(mit)
- music
- playwright-(emulator-automation)
- pop909-(mit)
- three.js
- training
- typescript
- uikit/uikitml
- vite
- web-audio-(audioworklet)
- webxr