-
-
high-level flowchart of our project, using 3 ML models tied together with DAIN + some algorithms + flask server
-
Quantitative difference metric between two pose sequences, error over the 1 exercise rep for each of 17 measured joints
-
GIF
gif of one rep of deadlift, result from pose estimation model
-
GIF
Another comparison of two deadlift pose sequences
-
GIF
Gif of qualitative comparison of pose sequences adjusted with dynamic time warping of ideal posture (blue) with user's unideal posture (red)
Inspiration
We were inspired to do this project as it not only let us implement machine learning models, which we enjoy learning about, but to tackle problems surrounding exercising and physical therapy.
What it does
Our model integrates with DAIN AI as a tool. People can send a video of themselves or someone else into DAIN AI. Our tool runs some python from our DAIN AI CLI that utilizing vision models will classify an exercise by what exercise it is, construct a pose (mapping joints and forming a skeleton) of the person in the video creating along with a person who is considering a golden standard and create an animation interpolating the two people's movement over each other creating a visual showing how ones form compares to a known good form, and calculate the error between the two poses returning both the raw data and it as a line graph. All these 3 things are returned in DAIN. (IMAGES and .gif hosted by us) DAIN can use the data and classification for users to ask more questions about their form.
How we built it
We built it using using a pipeline that got input images: a video of someone excursing and an image of someone with great form doing the same exercise.
Challenges we ran into
We had to pivot between models and data sets to combat generalization issues and perform better in video settings. Also interpolating movements between two poses with different rep speed was challenging. Training time was also long!
Accomplishments that we're proud of
We were able to put together more than one visual ML model (about 3.5 models), learn new algorithms like Dynamic time warping, understand why form tracking is so difficult, and we were able to integrate our ML / transformation logic into DAIN AI all in one day. We feel our application can have real implications on making something sports medicine/ health/ fitness knowledge and tools more accessible and easier to use.
What we learned
We learned about pose estimation (we had no experience with this), working with DAIN, interpolating time series data nonuniformly to minimize distance metrics (dynamic time warping), frontend dev (we also had no experience with this), using multiple AI models working together, curating data/inputs for analysis of an LLM. How to plan an build a project with zero preparation on very short notice!
What's next for FORM.AI
Create better models that can generalize better, produce more data, optimize transmutations across our code, and collect more data to include a broader range of movements
Log in or sign up for Devpost to join the conversation.