fitVision

Inspiration

During the end of the summer, we got together and brain stormed ideas to start our first project together. I personally wanted to do a computer vision project as I had just returned from a small computer vision research program at Cornell and recovered from a 2 year sports arm injury. However, we never really got the chance to start once classes quickly began, and this was the perfect opportunity for us to sit down and kick it off.

What it does

fitVision analyzes your bodyweight exercise form and provides live-feedback in video format on your form. It also counts repetitions!

How we built it

Our project uses the lightweight kinematic pose estimation model MoveNet single-pose thunder created by Google to detect landmark keypoints on a single person that can be ran on mobile applications (due to time constraints, we've ran the mobile model on a laptop). By using basic trigonometry to calculate angles between certain keypoints for a selected exercise in between its starting and ending positions, we are able to count the reps and provide live-feedback to the person's form when failure occurs. The latter is only a basic functionality of how far you completed the rep measured in degrees to a given threshold (green eclipses), but much more is planned for after the Hackathon.

Challenges we ran into

Besides for the couple hundred pushups performed all throughout the night (ouch), while calculating the angle between the vectors elbow->shoulder and elbow->wrist keypoints using the definition of the dot product was easy enough to count a repetition, we quickly found that it was easy to fake a rep by waving your hands in an upright position for the pushup exercise. To fix this, we calculated the shoulder->ankle and shoulder->wrist vector, though this proved to take more time to figure the correct thresholds and implement than realized.

Accomplishments that we're proud of

Going into this, we were happy enough to even successfully load the model and get keypoints visualized in a camera window. Although what we further accomplished isn't all that impressive in the grand scheme of things of what we plan to do with this technology, we are very satisfied with our efforts.

What we learned

We learned how to load lightweight .tflite models and interact with them to use as a tool towards a specific goal, which in this case was to analyze keypoints in an image to provide live exercise form feedback.

What's next for fitVision

As originally stated, the efforts made at the Hackathon are more or less meant to kickstart this project. Given our preliminary research, we believe there is an opportunity to test a calisthenics exercise form app in the current market as there are, surprisingly, not many alternatives for the average consumer. We believe many should be able to smartly and safely start their fitness vision even with minimal gym equipment while avoiding injury. Frontend coming soon.

How to run fitVision

See the GitHub link provided and the README.md. Very simple to setup and run!

Built With

• movenet
• python