Pose Estimator

Form check for squat in motion
Openpose on face & background

Inspiration

With the advancements in machine learning and computer vision, we wanted to see what we build without any previous knowledge, so we dived into the deep end by exploring its application on object detection. Specifically we wanted to see what a directly application would be like which inspired the idea to use pose detection to analyze the form of certain workouts.

What it does

Using the Twillio API, we take a video of someone doing a certain work out (like a squat), and upload it into the tf-pose program. The openpose library allows for a connection between the object detection from OpenCV and the ML power of Tensorflow leading to the pose estimator for the various body parts. These parts are treated as nodes and are used to calculate the approximate change in someone's movements, matching it with what is excepted for someone doing a workout. If there is a match, the program outputs the pose for the viewer and the program's accuracy in predicting if the workout has correct form.

How we built it

For front end, the Twillio API used flask to accept an mp4 file and message it to the program. In the back the ML library process the video and shows the resulting pose, this pose is used to calculate the form using an algorithm dedicated for each workout.

Challenges we ran into

We ran into a lot of difficulties as Tensorflow latest version (2.0) recently came out, making many of its dependencies and attributes incompatible with the openpose library, requiring extensive work in debugging issues with the set up of all the other programs. This lead to a time pressure toward the end as we were forced to rush the algorithm.

What's next for Pose Estimator

In the future we plan on expanding the accuracy of the program allowing the algorithm to be more dynamic. This can lead to numerous potential applications as multiple workouts or movements can be added.