Inspiration
As someone who has been doing research on spasticity and chronic pain relief for the past 3 years, I have talked to many patients who see physical therapists and are required to regularly exercise certain joints and muscles on their own as instructed by their therapist, to maintain a trajectory for speedy recovery. However, it is often difficult for patients to allocate time and stay consistent with these exercises. We’re all too familiar with life getting in the way of things, and in most cases, physical therapy is the easiest thing to push to the end of the priority list. But no matter how busy life gets, we’re never too busy to open up our favorite mobile game, scroll on instagram, hop on a late-night video game, or play a quick game of wordle. Thus, we thought that by combining gamification– an aspect of our lives that is becoming ever present in this digital age– with physical therapy, we can make rehabilitation more engaging, ensuring patients don’t stray from their recovery.
What it does
Thera_pixel offers two approaches to gamified physical therapy, one for younger and the other for older audiences. The first approach, Pizza Pursuit encourages patients to emulate upper extremity typically recommended by physical therapists to cook a pizza from rolling the dough all the way to cutting the slices. This game has a live video feed with ML pose detection accompanied by engaging and responsive pixel art visuals showcasing the pizza making process. Our second approach, Practice Studio, allows for patients to focus on perfecting the form and function for common physical therapy exercises, outfitted with real-time feedback to speed up or slow down the repetitions. Practice Studio is designed for older audiences to ensure they progressively overload their recovery safely and effectively.
How we built it
We built our project around the python CV2 library and Google’s MediaPipes 3d Pose Estimation tool, which allows us to determine relative positions of our joints in real time. Furthermore, we were able to analyze the accuracy of our movements using the IntelliRehabDS dataset, which we used to create ground truths progressions for the different movements.
Challenges we ran into
One major challenge we faced was finding an effective methodology to create our ground truths. We deliberated between different approaches including exploring learned models and measuring progress by positive changes in key angle values. Both of those approaches did not yield effective nor accurate results. We settled on incorporating a timer telling the user when to start a given motion, then give feedback on pacing, as at each clock cycle, the angle of the user's arm is compared to the expected angle according to the data at a time t after the timer starts. this seemed to work relatively well, as we were able to set it to a pace that is easy to follow along with and analyze in real time.
Accomplishments that we're proud of
Some big accomplishments we are proud of was our ability to take a novel, considerably small dataset and apply data science techniques like data cleaning and temporal interpolation to create consistent ground truth progressions for our desired movements.
What we learned
Through the development of thera_pixel, we gained valuable insights into the intersection of gamification, physical therapy, and motion tracking. We learned that real-time feedback and engaging visuals can significantly enhance user adherence to physical therapy exercises. Additionally, working with a small dataset challenged us to implement creative data science techniques such as data cleaning and temporal interpolation to generate reliable ground truth progressions. We also discovered the importance of balancing precision with user-friendliness, as overly strict tracking could discourage users rather than motivate them.
What's next for thera_pixel
Moving forward, we aim to refine and expand thera_pixel by improving the accuracy and adaptability of our motion tracking system. This includes training more sophisticated models that can better accommodate variations in user movement and different rehabilitation needs. We also plan to conduct user testing with both patients and physical therapists to gather feedback and tailor the experience for diverse demographics. Additionally, we envision expanding our library of gamified exercises, integrating more interactive elements, and possibly exploring augmented reality (AR) or virtual reality (VR) applications to further enhance engagement. Lastly, we will explore potential partnerships with healthcare providers to bring thera_pixel to real-world rehabilitation settings.
Log in or sign up for Devpost to join the conversation.