Inspiration
Learning piano is learning what your own two hands can do, and no screen can see your hands. A headset can. We wanted a first piano lesson where the teachers live on the hands they are teaching.
What it does
You sit at a keyboard. In front of you hangs a felt storybook window. Two characters live in it: Kiki, a quick hare who carries songs, and Boba, a steady tortoise who carries the low half. They leap out of the window and onto your hands. Kiki rides your right hand and plays the melody; Boba rides your left and plays the bass. Their story is that a song is two halves, and it only arrives when both of them carry it. So do your hands.
Every lesson is a short piece. A character hops onto the keys to show you how it goes, then rides your hand while you play. The keys glow where you play, and something in the window answers your music. Between chapters the two of them leap back into the window and the story moves on.
Hands are the whole interface. There are no controllers anywhere. You press keys with your fingers and touch buttons with a fingertip. Each character mirrors the hand it lives on, reacts when you touch it with your other hand, and a peace sign with both hands brings the piano and scene back to you.
How we built it
Unity 6 with the Meta XR SDK and hand tracking. Boba and Kiki are our own characters: modelled, rigged and animated in Blender from code, with a hand-written felt shader that also dresses the piano. The lesson engine grades pitch and timing, forgives near misses for beginners, and ramps tempo as you improve. Everything is seated and within arm's reach, and a lesson takes a couple of minutes.
Challenges we ran into
Making physical key presses reliable with tracked fingers, so resting hands do not trigger notes and a near miss still counts for a beginner. Keeping characters attached to hands that come and go from tracking. Staging a story in a window without ever taking the player's hands away from the keys.
Accomplishments that we're proud of
Original characters, story and melodies for a whole first book of lessons. Teachers that ride the hands they are teaching. A fully seated, hands-only experience where a lesson takes two minutes.
What we learned
When the hands are the interface, a character can teach by responding to them, and the lesson stops being a screen you watch.
What's next for Boba and Kiki
A window anchored to your real wall in passthrough, mapping to any real physical piano or keyboard, more chapters of the story, and a play-on-any-table mode.