Physio Hero (HTHS)

• 

  Main Menu

• 

  Instructions on playing the game

• 

  Choosing a part of the body to focus on

• 

  Arms minigame - "Arboreal Adventure"

• 

  Legs minigame - "Lofty Leap"

• 

  Torso minigame - "Tidal Trek"

• 

  Once finishing all reps of an exercise, you can choose another option

Team Members & School Name

This project was developed by Samvarth Raj, Rehan Ali, and Sebastian Postigo from High Technology High School (HTHS)

Inspiration

Physical therapy is one of the largest areas of healthcare. Doctors prescribe more than 50 million Americans with physical therapy annually (Harvard Health). Despite this, reluctance to do physical therapy is an increasingly growing problem, with 70% of physical therapy patients not completing their full course of care (NCDS).

Contrarily, Video Games are rapidly growing in popularity. There are currently 3.6 billion active video game players globally (Statista), and the market is growing by a compound rate ranging between 6.3% and 12.6% (Grand View Research).

Combining these two allows for Physical Therapy to become more accessible, fun, and engaging, reducing the reluctance for patients to practice it. Our application, Physio Hero, successfully does this.

Design-wise, Physio Hero was inspired by fitness video games, such as the Wii Fit series, Duck Life series, or Ring Fit Adventure. The short, burst-style minigame gameplay loop was inspired by the both WarioWare and Mario Party series, and the artstyle was inspired by cel-shaded games such as The Legend of Zelda: the Wind Waker, and Jet Set Radio.

What it does

Physio Hero uses Machine Learning and computer vision to track the reps of a given physical therapy exercise. This unique tool is presented in the form of a video game.

The game is divided into three main missions, which each feature a unique quest. The mission targeting your arms is a treetop swinging minigame called “Arboreal Adventure,” the mission targeting your legs is a cloud jumping minigame called “Lofty Leap,” and the mission targeting your torso is a swimming minigame called “Tidal Trek.” Each rep progresses you further in the mission by moving the character.

How we built it

For the game architecture, we chose to use an Entity-Component-System pattern using the Python library esper to cleanly distinguish between our separation of concerns. We found that it was much cleaner to develop using this pattern and allowed proper decoupling of different systems.

The core of the game, however, is seeing the player. This portion depends primarily on OpenCV and MediaPipe to detect repetitions of different physical therapy exercises and count them for our game logic. We used MediaPipe’s PoseLandmarker machine learning model to detect different joints and joint angles of each pose. In order to tell when the player was performing a repetition of a certain PT exercise, there were two ways we could have approached it; we could have hardcoded angle thresholds for each joint and stored a JSON for each exercise. However, we built a tool to automate recording angle vectors at certain keyframes (one for initial and the other for the peak of the movement). We used linear algebra to calculate a progress value between the initial movement and the peak movement. And when a certain progress threshold is reached, a rep is counted.

Challenges we ran into

Our two biggest challenges were tuning the machine learning model, and the time constraints.

This was our first time using MediaPipe’s Pose Landmarker in combination with OpenCV. This, paired with the time constraints, caused similar exercises to be confused with each other, for example, the lateral and parallel arm raises can be done for each other and still count as reps. The skeleton itself also glitches occasionally and gets confused if the player is not a certain distance away or if things are blocking the player.

As for the time constraints, we often found ourselves working for several hours a day without breaks due to the scale of Physio Hero, especially on Sunday (the final day). We were forced to scrap certain features, and modify others.

Accomplishments that we're proud of

We are proud of being able to finish such an ambitious project in the timeframe of just one weekend. Our timeboxing at the beginning of the process ensured we stayed on track each of the 3 days and didn’t have to deal with copious amounts of work at the last minute.

We are also proud of our implementation of the game portion of Physio Hero. We were able to use 3D tools in Blender to create animated spritesheets to use in PyGame, and the cell shader we created pulls together all the elements with a consistent cartoony art style, matching the minimalistic, energetic tone of our game.

What we learned

We learned how to utilize and interpret data from computer vision models. We also learned about different ways to handle tolerances for these models, both through differences in member angles and through progress towards the target pose.

We also gained deeper knowledge on PyGame, especially regarding handling spritesheets changing between animations based on the player’s state.

What's next for Physio Hero

We plan to expand the scope of the game by implementing additional physical therapy exercises for each target area.

We also plan to spend additional time tuning the tolerances of the model (MediaPipe Pose Landmarker) to both make detection more consistent and to allow for the game to respond to whether an exercise is done with perfect or poor form.

For better ease of use, we also intend to add an option to connect a phone as an external camera so it isn’t as difficult to stay in frame.

With these additions made, we’d like to publish the game to official platforms for widespread access. Currently, the PC build works, but we’d like to expand to Nintendo Switch 2, Mobile, and Web, as they all have built in cameras.

Built With

• esper
• mediapipe
• opencv
• pygame
• python