Input vs Annotated Corrected Output
Physical training is often quite expensive and without it, it can be difficult to understand the underlying muscle and bone position needed for optimal results, both for recovery and for improvement. Our plan was to automate the process of checking exercise form so that people would be confident they were getting the most out of their workout without the need for a physical trainer. To do this, I designed and built a webpage which allows you to submit a video of you completing simple exercises and returns the video of you overlayed with instructive feedback. Using flask, opencv, python, and Wrnch's API I developed this model by comparing the submitted videos against my own form and computing measure of deviance. Wrnch's API was fundamental because it gave us consistently accurate feature points. This allowed us to compute features for the correct form without having to detect joint positions. We then built a manifold of the correct form by using features computed from our form throughout the exercise. When the submitted video deviates from the correct form we use computer vision tools to indicate which parts of the lift are incorrect. This can be a difficult task because people may not always start their exercise at the same time as our ground truth or exercise in different locations. This required us to register the frames of the image across time so we could directly compare the features between our manifold and the given video. To circumvent the problem variance in the position we computed vector based features which allowed our network to work regardless of position and size. This is one of the advantages of our service over competitors. By registering across time and using vector based features we are robust to time differences, spatial differences, and positional differences. Integrating and learning Wrnch's API helped us quickly prototype and fine tune our model and exploring advanced computer vision techniques gave us inspiration for our own model and information about the challenges facing augmented reality.
Challenges I ran into
The main challenge provided was with respect to Wrnch's API. Our transformation matrices had been accidentally misaligned with respect to their coordinate system and required us to undergo extensive debugging that cost our team a lot of time. Recreating videos from the decomposed frames were also troubling, especially when integrating with flask as the codec has some compatibility issues.
Accomplishments that I'm proud of
We are proud to have an introductory demo ready with a robust algorithm with large potential to upscale with little overhead. This was our teams first hackathon and we are proud to have been able to make it to the end.
What I learned
We learned how to further implement computer vision techniques without the need to train large scale deep learning models. Having the granularity with our algorithm will allow us to further expand in specific areas we see fit while maintaining interpretability.
What's next for FormCheckr
We would like to expand FormCheckr for more exercises and a real time API. Deep Learning in particular would help with expanding the variety of exercises we could provide service for and improve our feedback.