Are you struggling to motivate yourself to go to the gym? Is doing the same repetitive exercises over and over too tedious at home? Is COVID ruining your chances of getting swole? Welcome to WeFit, an immersive web application that motivates individuals to stay fit through the gamification of exercise.
What it does
In our app, raindrops fall from the sky. The goal of the game is to collect the raindrops to prevent them from hitting the floor. And to collect a raindrop, you need to do the exercise indicated by the colour of the raindrop.
How we built it
We collected real-time video data from a webcam through WebRTC APIs.To detect if a proper pushup, situp, or squat was completed, we leverage a field of Deep Learning called Pose Estimation to detect what pose orientation a user is in. We utilized Posenet, a TensorFlow library that predicts your body position in real-time using a Residual Neural Net. We then used Node.js/Express.js on the backend to host our frontend code during the deployment phase. In the future, we can also store user calibration data and other personal stats. A lot of work is done on the frontend. We used React.js to render the video, add rain drop animations, and create the game logic. Finally, we did some HTML/CSS magic to make the website look AMAZING.
Challenges we ran into
Some initial challenges we ran into include rendering the live stream using React, and properly overlaying the canvas with the computed pose during realtime using Tensorflow's Posenet. We had extreme lag during displays. Merging conflicts. Errors with versions of node modules. Animations in react, proper estimation and exercise repetition tracking. And recently video and audio editing was a hassle.
Accomplishments that we're proud of
We're proud that we got a working demo, and a clean landing and game page. It's great that we also got the PoseNet to overlay correctly on the video, and track movements in realtime. It took a while to get a working algorithm that tracks rep completion, but we're happy that it was finished as well. Overall, we're satisfied that we were able to completely flesh out an idea that helps improve fitness motivation during quarantine (especially with the new restrictions happening) within 36 hours while learning new technologies and dealing with video submissions. Also, our video intro is fire.
What we learned
We learned how to connect React with WebRTC APIs to collect video data. We also learned how to incorporate Tensorflow.js libraries in our web application. Using machine learning models for real-time pose prediction was new and spicy stuff for us, as well as exploring different animation libraries in React, before ultimately deciding to build some custom components ourselves.
What's next for WeFit
We plan to implement multiplayer mode, add more exercises, and generalize our model to further improve exercise prediction accuracy.