Inspiration

Drums are the instrument everyone wants to try and almost nobody can. A kit is expensive, takes up a room and is loud enough to end friendships with your neighbors. But everybody already drums: on the table, on the steering wheel, on their knees.

That gave me the idea for CapyDrummer: Hands-On Beats. What if the table you already tap on became a real drum kit, your bare hands became the drumsticks, and the most relaxed animal on earth taught you how to play?

Hands-first was never an extra for this project. It is the game. Drumming is maybe the most natural thing a hand can do, so it felt like the perfect way to show what hand tracking can feel like when it is designed for, not added on.

What it does

You sit at your own real table and trace its outline with one finger. A drum kit builds itself onto it, part by part, like a little transformer. Then you play with your bare hands:

  • Fingertips or a flat hand hit the snare, toms, hi-hat, ride and crash.
  • A fist tapped on the kick pad is the bass drum.
  • Capy, a round low-poly capybara, teaches you, plays along on his own tiny kit and cheers you on. He only says "perfect" when you actually earned it.

There are 17 original songs with a full band, from beginner (quarter notes, two drums) to advanced (syncopation, funk, 128 BPM). Each one teaches a single idea: the backbeat, rests, fills, accents, a 3/4 waltz. A Note Highway sends glowing gems down straight lanes that land on each drum exactly when you should hit it.

And your room joins the show: a heavy kick sends a wave of light through your real walls, floor and furniture using the headset's room scan, and the final hit of a song sets off fireworks in your living room.

Built for the competition: hands-first, seated, quick to start

Competition goal How CapyDrummer meets it
Fully usable with hands, end to end Every single step works with hands: intro, table setup, tutorial, songs, menus, pause, the closet and the kit builder. Controllers are optional and become two virtual drumsticks, but the home menu reminds you that the game is made for hands.
Seated play You sit at a table. Everything, even the giant power-up drums, stays within reach of a seated player. No roomscale, no standing up.
Fast cold start The table is remembered between sessions. Coming back takes a few taps to confirm it.
Clean pause and resume Poke Capy (or use the Quest hand-menu gesture) and the song pauses gently, offering Slow, Song tutorial, Home or Close. If you stop playing, the music fades and Capy asks what's up.
Satisfying within 10 minutes The first run is one continuous lesson that ends straight in your first full song. No wall of text, no menu first.
Ready for Meta VR Glasses No controllers, seated, short sessions, passthrough first. It is the kind of experience the Glasses are made for.

Hand interactions everywhere, not just on the drums

  • The opening scene is a hand moment. A branch grows out of your ceiling, Capy hangs from it by one paw, and asks you to hold out your open palm. He lets go and drops into your hand, riding along as you move it, then hops onto the table. In the first thirty seconds you learn the core idea: your real hands touch things in your room.
  • Capy is touchable. Poke, boop, pat his head, tickle him or give him a high five, and he reacts to each one differently. Offer a flat hand in a calm moment and he hops on.
  • Hidden gestures. Throw a rock sign and Capy pulls out a guitar for a solo. Make a heart with both hands and he dances. Finding them unlocks outfit items early.
  • Direct manipulation. In the garage you rearrange your drum layout with your hands: pinch or grab a drum with a fist and move it wherever it feels right for you.
  • Your hands are part of the show. Seven hand looks (Magma cracks glow hotter with your combo, Galaxy, Crystal, Liquid Gold and more) are drawn on your real tracked hands, plus finger trails like Comet, Stardust or little floating notes.

How I built it

  • Engine: Unity 6, URP, OpenXR and the Meta XR SDK, for Meta Quest 3 and 3S.
  • Mixed reality: passthrough first, with Meta MR Utility Kit and the room mesh. Real hands always render in front of the virtual drums.
  • Table setup: the finger-traced outline is fitted to a level rectangle and snapped onto the scanned table surface.
  • Audio: six genre drum kits (studio, vintage, rock, funk, lo-fi, electro) with several samples per drum, alternated so repeats never sound robotic. Every drum is tuned for the Quest's built-in speakers.
  • Music: 17 original songs, each with a slower version for practice.
  • Capy: fully voiced with over 700 lines (ElevenLabs), lip-synced, and procedurally animated: breathing, blinking, ear twitches, nodding to the beat, stick flips, yawns and sneezes.

Challenges I ran into

1. Zero-lag drumming with bare hands: my biggest win. Hand tracking always runs a little behind your real hand, and for drums that is usually a dealbreaker. I beat it. CapyDrummer does not wait for the finger to land: it reads a committed downstroke and predicts the exact moment of impact from the fingertip's height above the drum head and its speed, roughly

$$t_{hit} \approx t_{now} + \frac{h}{v}$$

and fires the sound right on that moment. The result: the drum sounds the instant your finger touches it. It feels like real drumming, not like hand tracking. This is the breakthrough that makes the whole game possible, and the thing I am proudest of.

2. Timing should feel fair without ever cheating. In songs every hit is pulled gently onto the beat, onto its note if there is one, otherwise onto the beat grid. One strict rule: a sound may be held back a moment, but it is never played earlier than you hit. That keeps it feeling like your playing, just tighter.

3. Tracking glitches turned into ghost hits. A wrist that flips in one frame or fingers that suddenly scramble would trigger random drums. I added a filter that rejects any hand pose a real hand cannot make, so glitches stay silent.

4. A fist as a kick drum. Telling a deliberate fist tap apart from a hand that just happens to be closed took a lot of tuning. It is now one of the moves players like most.

5. Bass from tiny speakers. The Quest's built-in speakers cannot play deep bass. The kick drums are recorded sound effects shaped so their punch comes through anyway.

6. Helping beginners without playing for them. When you struggle, the timing windows widen and upcoming patterns quietly get easier. When you play well again, it goes back to normal. The game never plays the notes for you, so success always feels earned.

Accomplishments I'm proud of

  • Drumming with bare hands that actually feels like drumming, not like waiting for hand tracking.
  • Someone who has never touched a drum goes from zero to a full song in a few minutes.
  • Capy turned into a real character: honest, goofy, slightly sarcastic, and he never stands on your note lanes.
  • The kick drum lighting up the whole real room still makes me smile every time.

What I learned

  • Hands-first is a design decision, not an input setting. Every interaction (pause, menus, customization, even the intro) had to be rethought for hands instead of mapped from buttons.
  • Latency is a game design problem. Prediction and gentle timing correction mattered more for the feel than any visual effect.
  • Honest feedback motivates more than constant praise. Players care about Capy's "perfect" because he does not hand it out for free.
  • Mixed reality is strongest when the room reacts. Light washing over real furniture sells the magic better than anything floating in the air.

What's next for CapyDrummer: Hands-On Beats

  • More songs and new genres
  • A free play mode and recording your own beats
  • Playing along with friends in the same room
  • Bringing CapyDrummer to Meta VR Glasses at launch: seated, controller-free and short-session by design, it is ready for them from day one.

Built With

  • c#
  • elevenlabs
  • hand-tracking
  • meta-quest-3
  • meta-quest-3s
  • meta-xr-sdk
  • mixed-reality
  • mr-utility-kit
  • openxr
  • passthrough
  • procedural-animation
  • scene-understanding
  • shader-graph
  • spatial-audio
  • unity
  • urp
  • vfx-graph
Share this project:

Updates

Submission history