Amidst the COVID-19 global pandemic, social distancing and physical isolation has brought an increased emphasis on mental health and physical health being a ways of coping with these challenging times. According to a recent study done by CTV, about 16% of Canadians said their mental health is worse or somewhat worse. Consequently, there is a need for accessible free physical fitness resources for those who need it.

The goal of this project is mainly to provide an accurate and reliable alternative to users who may want to learn yoga from home with real-time feedback.

What it does

The project simulates a real-time yoga instructor that provides an accurate and reliable yoga pose classifications in with real-time feedback. The machine learning model classifies 4 different yoga poses and provides a report of the results based on the user’s current yoga session.

How we built it

To understand how the image classification and PoseNet model would work in sync, it was conceptualized that there were 7 essential rudimentary steps towards the final model. They are as follows: Step 1: Load or create some yoga poses and store them in .json format. Step 2: Set up and initialize neural network options. Step 3: Add data to the neural network. Step 4: Normalize the data. Step 5: Train the neural network. Step 6: Use the trained model to make a classification. Step 7: Present the results.

Challenges we ran into

Performance testing of the classifier was performed extensively for all four poses, namely, mountain pose, tree pose, goddess pose and warrior 2 pose. For each of the four poses, testing was performed under four lighting conditions – dark, dim light, medium light and bright light. Along with this, for each of the four poses, frame rates were tested for 30 fps, 45 fps, 60 fps and 100 fps. Tests were carried out by multiple people of different heights and weights in order to achieve realistic results since the classifier was trained on a similar data set, the size of the people performing the pose had a negligible effect and an aggregate was taken for the calculations. The classifier performed best for the mountain pose.

For the Goddess pose, the system did not detect the pose at all and hence the accuracy score was approximately 0.00% for all testing performed.

For the Warrior 2 pose, the system performed best for moderate to bright lighting conditions and the frame rate had a minimal on the aggregate accuracy. The system was also unable to detect the key joints at all and hence the accuracy score was approximately 0.00% for all testing performed.

The overall performance of the model is quite respectable, considering the model was validated against a new fresh set of data points. Using angles computed from the raw data points from PoseNet improved the accuracy and performance of the model greatly. When a user selects and performs a yoga pose, the accuracy of the user’s model is measured as the ratio of computed angles that fall within an acceptable range of errors over all angles that are deemed unacceptable.

Accomplishments that we're proud of

The website was successfully deployed using Github Pages once the usability of the site was tested systematically. The classifier detected the yoga performer accurately once the timer started. The data was collected live and accuracy was calculated with every movement. The average accuracy was calculated and displayed at the end of each session for each pose.

What we learnt

The capstone project allowed for developing an end-to-end Computer Vision based AI for Human Pose Detection. The completion of which would not have been possible without four different phases each led by a different Project Manager, PM. The first phase was where the Machine Learning Model was developed led by Soumya Chaudhari. The major outcomes of this phase was a working pose estimation model, image classification using test cases and a working loss function of the classifier model. The second phase was led by Vatsal Shreekant where the deliverables were the final image classification model, prototype of the UI and metrics associated with the model. The third phase was led by Vincent Yeung where the deliverables were results of performance testing(covering different scenarios) and demo of the website with additional features such as scoring system and user authentication, etc. The fourth and final phase was led by Dimple Gamnani. The deliverables in this phase were to perform system testing on the model, deliver the final website and a sample reports page.

What's next for Computer Vision Based AI for Human Pose Detection

For future improvements we hope to have a well-defined database using MySQL and MongoDB. We hope to host our website on another platform other than GitHub pages hence, looking into hosting the application on Amazon Web Services (AWS) or Google Cloud Platform (GCP). As well, allow for poses in the dark to be detected, ability for users to download their yoga history, user login system and add more poses to the model. Finally, we would need a better feedback system that tells the user exactly what poses to fix in addition to the red marks on the skeleton that show up on the canvas.

Built With

Share this project: