PosePal

Real-time pose feedback for better recovery.

Comment

Inspiration

Physical therapy is inefficient for both patients and therapists. Patients’ rehabilitation poses are not properly evaluated, and the mistakes they make can delay their recovery for weeks, while therapists are overwhelmed with the number of patients they have to handle.

I observed this firsthand when I ran cross country in high school. I used to suffer a lot from shin splints and other leg-related problems, which required the need of physical therapy. I noticed that many of these PT sessions were unnecessarily held up by patients doing exercises that they could just as easily do from the comfort of their homes. This gave us the primary inspiration to make an app that alleviates the strain on PT facilities, giving them the time to focus on actual rehabilitation.

What it does

PosePal solves this problem for both patients and therapists; therapists can fluidly create poses and assign them to patients, specifying max daily sessions and due dates to make physical therapy clear and concise. On the other hand, patients are evaluated frame-by-frame as they attempt poses, with our machine learning model pinpointing accuracy and timestamps that inform them of mistakes.

How we built it

We trained our pose analysis model by freezing the first few layers of the MoveNet Lightning model and then training it further on a Kaggle dataset of 5 key rehabilitation poses. This way, the model retains the vast amount of pose data it already has while improving its applicability to our use case.

We utilized AWS CloudFront to create a signed URL that expires after 24 hours for any interaction that attempts to access video resources to ensure patient privacy. The video footage itself was stored in Amazon S3 buckets for higher performance and scalability.

We handled authentication with Supabase Auth, with patients pairing with therapists using time-to-live pairing codes that maximize privacy, as required by HIPAA.

AI Usage We utilized Lovable to create an initial wireframe of the front-end structure. In addition, we utilized Claude for prototyping segments of the code base and identifying sources of key errors.

Challenges we ran into

The testing data for rehabilitation poses were rather limited, so we could not expand the pose set as widely as we desired. In the future, we may incorporate the processing of input poses by licensed therapists with appropriate permission to enhance the breadth of the model.

Accomplishments that we're proud of

Our model ran with low latency and high responsiveness to movement, while accurately identifying timestamps in videos that had mistakes. Also, configuring CloudFront and utilizing S3 ensures our project’s scalability and security to benefit patients.

What we learned

We learned how to fine-tune a pre-trained pose estimation model for a domain-specific use case, in this case, researching the best models that reduce latency on the frontend while being accurate in pose estimation. On the infrastructure side, we deepened our understanding of building privacy-conscious systems, from time-limited pairing codes to signed CloudFront URLs that protect sensitive patient video data. We also gained experience bridging the gap between ML inference and a real-time web interface, running TensorFlow.js in-browser with WebGL acceleration to keep latency low without a round-trip to the server.

What's next for PosePal

The most immediate priority is expanding our pose library. Right now we support five key rehabilitation poses, and we plan to work with licensed physical therapists to label a broader dataset covering various muscle groups. We also want to introduce a progress dashboard for therapists, giving them visibility into how each patient is improving across sessions rather than relying on patient self-reporting. On the patient side, we're exploring real-time audio feedback during live sessions so users don't have to break form to check a screen.

Built With

Share this project:

Updates