Make with Puppet

Thumbnail
Control a puppet using hand tracking.
Strike the specified poses with the puppet to successfully perform the play.
Perform a play based on the Japanese folktale Momotaro.
For example, this is the pose for defeating the ogre.
At the end, you can watch highlights of all the poses you made.

Inspiration

We wanted to create an XR game where hand tracking is not just an alternative to controllers, but the core of the gameplay itself.

We were especially interested in the awkward but fun feeling of controlling something with your own fingers. You know what you want the puppet to do, but making it move exactly as intended is surprisingly difficult. We thought that this slight frustration could become part of the fun.

From there, we combined puppet manipulation with short, fast-paced challenges: players see a target pose and must quickly recreate it with the puppet.

What it does

Make with Puppet is a hand-tracking XR game where you control a puppet, match poses, and perform on stage.

Players move the puppet directly with their fingers and try to complete different poses within a time limit. As the game progresses, more fingers become available, gradually increasing the difficulty.

The player's performance is also shown like a TV stage show, so people watching outside the headset can enjoy the experience together.

How we built it

We built the game around hand tracking from the beginning.

Instead of mapping hand tracking onto interactions that could also be done with controllers, we designed the puppet so that each finger directly affects its movement. This makes the player's own hand motions an essential part of the experience.

One important part of the design was making the game understandable for first-time XR players. Although hand tracking gives players a lot of freedom, too much freedom can also be confusing.

To solve this, we:

focused the gameplay on one simple objective: match the target pose
gradually increased the number of fingers players could use
used a familiar puppet-show setting to make the interaction easy to understand
presented the gameplay as a stage show so spectators could follow what was happening

Challenges

Our biggest challenge was balancing freedom and accessibility.

Using all ten fingers provides many possibilities, but it also creates a high learning cost, especially at an event where players may only have a few minutes to understand the game.

We learned that adding more interactions was not always the answer. Instead, we simplified the objective and introduced the controls step by step.

Another challenge was making XR enjoyable for people who were not wearing the headset. XR can easily become an isolated experience where only the player understands what is happening. By presenting the gameplay as a TV-style performance, we made the player's actions, successes, and funny mistakes visible to everyone around them.

What we learned

The biggest lesson was that limitations can make a new interaction method more approachable.

Hand tracking gives players a huge amount of freedom, but carefully restricting that freedom can make the experience easier to understand and more fun.

We also learned how important spectators are for an XR experience. At BitSummit, more than 80 people played the game, including many first-time XR users, and we saw families and groups enjoying both playing and watching together.

Make with Puppet ultimately received the XR Category Award and two Jury Awards at BitSummit Game Jam 2026, which gave us confidence that hand tracking can become more than an input method—it can become the game itself.