https://github.com/raincrash/LeapFit

Inspiration

Physiotherapy is scientifically proven to be one of the most effective ways to treat and prevent pain and injury. Studies have explored four themes that may influence patient-therapist interactions: interpersonal and communication skills, practical skills, individualized patient-centered care, and organizational and environmental factors. This project aims to address a part of the problem by motivating a patient to try out therapy right from the convenience of home via a mixed reality device.

The aim of the application is the motivate individuals to exercise right from their homes. The experience is gamified with an energy meter filling out based on how well the individual mimics the coach's therapy poses.

  1. The application runs right from the browser and supports MagicLeap's Helio browser for a personalized mixed reality experience. The avatars rendered are personalized to each individual.

  2. The application is generic to use cases and can be easily extended. It does not depend on the headset or the head poses and runs right from a browser.

What it does

An individual logs in to the account via a browser. Then a demo video of a coach performing certain exercises is shown. The individual tries to mimic the postures shown in the video and an energy meter fills up as the moves get closer. Once a set is done, the individual can visualize the sequence of poses in a mixed reality environment and correct themselves. The poses and shape of the individual are captured during the session and the avatar is customized.

How we built it

Web client: A native javascript application runs the demo video and a webcam, along with the visualization of the energy bar. A pre-trained mobilenetv2 (tensorflow.js) is used to get capture the poses. The frames are synced between the webcam and the video for a fair score. Our custom loss function gives out a score based on the match. The matched frames and the scores are sent to the server.

Server: A nodeJS based hook triggers a python build system running an instance of Human Mesh Recovery that estimates the SMPL parameters (10 for shape and 72 for pose) of a human deformable model based on the input frames. This deformable model is custom to each individual's shape and pose. Then this model is saved into a Prismatic/WebXR readable format (GLB) and cached on the server.

Helio Browser hook: The server triggers a Prismatic/WebXR render that can be visualized on a mixed reality headset and the user can pull the models out from the browser and visualize.

Challenges we ran into

This is our first mixed reality project. Setting up and playing around with MagicLeap's browser-based rendering was challenging since the WebXR support is still in beta.

Accomplishments that we're proud of

We formed a team and came up with the idea during the hackathon and making an end-to-end flow work in 24 hours is an accomplishment by itself. :P

What we learned

File format conversions are always challenging. Mixed reality is an exciting field and it's applications as a health coach is tremendous.

What's next for LeapFit

Release it as a fully-fledged web application. Spatio-temporal smoothing of the GLB animation instead of frame-by-frame tracking.

Built With

Share this project:

Updates