-
-
Introduction
-
Excerise 1
-
Excersize 2
-
Work those legs!
-
Complex movements!
-
Full Body Tracking built in!
Inspiration
Movement and sports have always been within my interests. With the new Pico Motion Trackers and Full Body Tracking it is possible to do complex movements with your body. I always like to help others move better and become more fit, so I wanted to share my knowledge and enthusiasm with everyone using XR body tracking! This means you can train muscle groups you cannot train with other VR workout apps.
What it does
Full Body Fitness VR is a personal trainer fitness app using full body tracking and AR. This changes the focus from boxing and simple movements in VR to more complex movements. There are exercises in the demo that will focus on core strength, leg strength, and balance, which are not possible without full body tracking. Examples are 1 leg balancing, side leg raises, lunges, and complex balancing exercises. It uses AR passthrough to allow people to train in a safe environment they are familiar with, lowering the threshold to start a workout! Spatial anchors could be used to have users place the personal trainer and their own positions.
How we built it
I used Unity 2022.3 with the Universal Render Pipeline to get everything to work. I recorded a personal training session with a green screen, cut up the video, and imported it inside of Unity. Here I use a custom green screen shader to cut out my body. I use my custom Shadow Receiver URP shader to catch the shadow from the body onto the real ground, to make the scene more immersive. With a testing app, I can record body poses, which I can combine with the different movements I do on the video to make recognizable poses that can be tracked at runtime. The pose I recorded beforehand will be compared to the current pose of the user, and based on that you will complete exercises one at a time. All the poses are stored in scriptable objects, making them very easy to manage in their own files. The sensitivity of each joint can be adjusted here with sliders, to make the setup very easy. All objects come together in a list (containing workout element structs), which I loop through when the workout plays.
Challenges we ran into
Full body tracking is still very new and does not have in-depth documentation. This means I had to learn how to properly utilize full body tracking. Later on, I looked into the full body tracking demo on the Pico GitHub, which helped me get the user's virtual body rendered properly.
Video playback in Unity is very limited, which limited my options on the use of transparency and framerate. This was fixed after 2 days of exporting to different video formats and testing. I had to downgrade from 4K 60 fps footage to 1080p 30 fps footage. Using video tools as AVPro in the future can fix this and allow greater quality video playback.
Accomplishments that we're proud of
To my knowledge, this is the first VR/XR workout game that can make use of full body tracking. Having a working prototype next to the concept itself is something I am very proud of and can't wait to show to more people, The use of full body tracking is also something that was new, so I am proud to showcase this to the public as well.
I am also pleased to see my virtual personal trainer video to appear quite naturally in AR, without doing 3D animation. The custom shadow on the ground can really help bring the scene to life and increase immersion. This is the first time I used AR on an HMD and I am happy with the end result.
What we learned
The biggest point I learned is that a flexible workflow is key. It took a lot of time to get full body tracking working, but because the systems I used are set up to be flexible, it was quite easy to put together a workout inside of Unity, based on just video and the recorded poses. Since I was sick for over a week, having this flexible backend was crucial for finishing the pre-release. Learning to use AR on an HMD was great as well. The Pico tools were new as I have only used OpenXR in the past. I was positively surprised by the ease of use of the Pico SDK. I also learned that there are limitations within Unity's included video player, which is something I will take into account early on for new projects. Right now this took a lot of time to get right.
What's next for Full Body Fitness VR
I first want to get the opinion of my peers on the app. I want to demo the application to multiple people to see how well it works and how well the tracking can keep up. If the response is positive I want to refactor the code to include multiple different workouts, as well as improve the tracking quality of the recorded poses compared to the user poses. This will be done with a quick calibration inside of the application. Once that's done a more professional recording setup will be used to record different workouts. Giving people options on what to train.
I also want to integrate hand tracking. Hand tracking with full body tracking should be supported in Pico OS 5.9 according to the contact I've had with Pico. This could mean that all workouts are possible, including pushups, situps, squads, and maybe even calisthenics exercises, like handstands
Log in or sign up for Devpost to join the conversation.