Inspiration
Blending the real environment with virtual objects is the key feature of mixed reality (MR). Since hand-tracking works really well with the Quest 3 we saw the opportunity to control virtual player characters in the real enviroment using fingermovement and gestures. We got inspired by the hand physics lab and the hand called "Thing" from the Addams family, which also moves around on its own. Gestures provide a great way of adding intuitive controls to a player character, rather than mapping interactions to abstract buttons. That is also where we saw the opportunity to "spice" existing hand-tracked walking demos up. Mixed with the scene understanding provided in the Meta-XR SDK we saw the opportunity to make a cool parkour game that makes your real room the playground.
What it does
The game starts with a building phase and is followed by a playing phase. Once you open the application you can build your own creative parkour level. This means the real physical world is your level and you can place the content (like lava, ladders, ziplines, start and finish) in it using spatial anchors. After the level is built, you can play through level yourself, or give your headset to the people around you to challenge them to a solution.
Your player character is an animated hand. Once the level is built, you can move your the virtual hand on the ground, by moving your actual fingers! But you can not only move, you can also make gestures that help you to interact with the environment. Just do a pinching gesture and slide down a virtual zipline from one table to another and overcome the lava on the ground! This provides a fun and intuitive interaction which is very responsive and at the same time provides a lot of identification with the player character, which is your hand. Find your way past all obstacles and reach the end with creative movements.
How we built it
We used Unity (2022.3LTS) with the Universal Render Pipeline to set up the project. We used the latest available tools, including the Meta SDK with building blocks. We used this to quickly set up the game scene, in which we added our gameplay.
For the hand movement, we made a new hand mesh with complete physics. We record the movement of each finger and use this movement to make the hand move, as well as to recognize gestures as interactions!
The level building phase makes use of the spatial mesh of the real environment and locally stored spatial anchors to place the virtual content elements.
Challenges we ran into
The biggest challenge was making a remote-controlled hand on the ground. This was done early on so we had enough time to polish up the interaction.
Having the hand zip along the zipline was also a hard challenge, since we place the ziplines dynamically. In the end this was done with a quick solution by linking together multiple components in the scene.
The last big challenge was rotating the hand. Doing this with the dominant hand (which already moves the hand) did not feel intuitive. We make the virtual hand grabbable, so you can simply grab the virtual hand with your real ones!
Accomplishments that we're proud of
Developing an interaction that has not been done before in this way is something we are very proud of. People who experience this for the first time are amazed and find the interaction fun!
Having such a complex interaction, that still is fun to play, is something definitely nice.
What we learned
Having realtime physics on dynamic objects, like hands, is very hard. We are happy that we tackled these problems early on and learned how to work with this on the go.
Getting good control over the spatial anchors
Producing a game with a working game loop in MR with very sophisticated interactions in only two days!
Getting a better feeling of rapid prototyping processes in a team of 5 (awesome) strangers!
What's next for HandWalker
As the app currently stands the app is asynchronous multiplayer. One person can make a level, and pass the level to a friend who can go through it. This means you can bring together people, which is good to see. Making this multiplayer (every users has an own hand they move) and having the spatial anchors transfer between play sessions would be amazing, so you have a library of levels you can easily switch between levels in your surroundings and play it with your friends.
We also want to:
- polish the locomotion of the player hand
- include more gestures for interactions -add more obstacles and traversal methods in the level builder
- Polishing the level builder
- Graphics touchup and consistent design
- Sound enhancements
- Add a menu and selection for already created levels
Log in or sign up for Devpost to join the conversation.