Inspiration
During the COVID-19 pandemic, we noticed that many people were leading unhealthy lifestyles and not exercising since many facilities such as gyms were closed. This inspired us to create a cost-free fitness platform which encourages people of all ages to pursue healthy active living from their own homes, without the need to hire an expensive personal trainer.
What it does
Sensai allows you to choose one of our custom-made recommended workouts, each of which has a routine of exercises and reps. Using an integrated webcam combined with MediaPipe’s AI pose estimation, Sensai tracks your progress by automatically counting the number of reps you do, while alerting you if your form can be improved and giving you hints to improve your technique through both text-to-speech and visible dialogs.
How we built it
The backend API consists of a Flask webserver that handles authentication, image processing, and workouts. We used JWT (Json Web Tokens) for authentication, and MongoDB as our database. It also uses SocketIO for real-time data exchange with the client.
The AI was created in Python using MediaPipe's Pose Model, which finds (x, y) coordinates of body landmarks.
The frontend was built with NuxtJS and WindiCSS, and designed on Figma.
Challenges we ran into
One challenge we faced was getting the AI to count the number of reps by detecting the change in angle between certain joints during the exercise. We needed to identify optimal angle ranges for each exercise, which caused a process of trial-and-error. Another challenge we faced was integrating the AI with the frontend. The logic was complex, so we had to draw diagrams and organize all our thoughts in notes to map out how the code would work.
Accomplishments that we're proud of
We are proud of that we were able to create the AI using Python and MediaPipe, despite using both for the first time. It was also our first time using OpenCV, which was used to locally test the AI without integrating it with the frontend. We also feel accomplished that we were able to think of a solution of calculating the angle between the joints/landmarks, and counting the number of reps by determining the change in the user's state from "down" to "up". Furthermore, we are proud that we were able to develop a complex, custom frontend interface to the AI using websockets for real-time data sharing of the webcam frames.
What we learned
We learned how to use MediaPipe to extract pose landmarks and use them for calculating angles to count reps and provide form hints. We also learned how to use SocketIO to stream webcam frames to the server in a real-time, low-latency fashion.
What's next for Sensai
As for what’s next for Sensai, we have much planned for its future. We hope to expand our library of workouts and include the option for users to create their personalized workouts. Improving the experience of the user, we also plan to implement features that focus on motivating the user such as streaks and achievements. We also hope to implement a screen in between exercises that enables users to reset their space before continuing with the workout. Another feature we have planned are group sessions, where users can call with friends and workout together.
Log in or sign up for Devpost to join the conversation.