Physiotherapy is scientifically proven to be one of the most effective ways to treat and prevent pain and injury. Studies have explored four themes that may influence patient-therapist interactions: interpersonal and communication skills, practical skills, individualized patient-centered care, and organizational and environmental factors. This project aims to address a part of the problem by motivating a patient to try out therapy right from the convenience of home via a mixed reality device.
The aim of the application is the motivate individuals to exercise right from their homes. The experience is gamified with an energy meter filling out based on how well the individual mimics the coach's therapy poses.
The application runs right from the browser and supports MagicLeap's Helio browser for a personalized mixed reality experience. The avatars rendered are personalized to each individual.
The application is generic to use cases and can be easily extended. It does not depend on the headset or the head poses and runs right from a browser.
What it does
An individual logs in to the account via a browser. Then a demo video of a coach performing certain exercises is shown. The individual tries to mimic the postures shown in the video and an energy meter fills up as the moves get closer. Once a set is done, the individual can visualize the sequence of poses in a mixed reality environment and correct themselves. The poses and shape of the individual are captured during the session and the avatar is customized.
How we built it
Server: A nodeJS based hook triggers a python build system running an instance of Human Mesh Recovery that estimates the SMPL parameters (10 for shape and 72 for pose) of a human deformable model based on the input frames. This deformable model is custom to each individual's shape and pose. Then this model is saved into a Prismatic/WebXR readable format (GLB) and cached on the server.
Helio Browser hook: The server triggers a Prismatic/WebXR render that can be visualized on a mixed reality headset and the user can pull the models out from the browser and visualize.
Challenges we ran into
This is our first mixed reality project. Setting up and playing around with MagicLeap's browser-based rendering was challenging since the WebXR support is still in beta.
Accomplishments that we're proud of
We formed a team and came up with the idea during the hackathon and making an end-to-end flow work in 24 hours is an accomplishment by itself. :P
What we learned
File format conversions are always challenging. Mixed reality is an exciting field and it's applications as a health coach is tremendous.
What's next for LeapFit
Release it as a fully-fledged web application. Spatio-temporal smoothing of the GLB animation instead of frame-by-frame tracking.