The benefits of exercise are clear from a mental and physical standpoint, we have all heard about them. However, during a time of social distancing and the pandemic, there’s been a growing apathy to exercise at home with energy and intensity. A lot of this is due to a distinct lack of external motivation from others to collectively work together to push each other to new heights. We needed to bring back collectivism and “gamify” at-home exercises. This is where Avastha has joined the lobby.
Fun fact: The name Avastha is Sanskrit for state or position. We initially wanted to call the project “Pose it!”, but it felt a bit like a ripoff of the “Bop It!” games.
What it does
- Avastha adds a competitional component to exercising.
- Users can create and join rooms of up to 4 people. The customizability of the room creation feature includes room name, duration, level of difficulty, and an option to keep/remove music.
- In a room, a pose will be shown for the user to mimic to the best of their abilities.
- Our machine learning algorithm will judge the accuracy of the pose performed and award points at an interval of (undefined).
- During the game, players can see the scores change and at the end of the game, final scores will be displayed and uploaded to a leaderboard.
How we built it
Frontend: HTML/CSS/JS was used for our website frontend. Tailwind 2 was the CSS framework we implemented to have a unique and nice design.
ML Model: Firstly we created the dataset which we specifically required for the poses. We trained over 600 images of yoga poses for our model built using 2 libraries called ML5 Js and P5js. We gained 96% accuracy on our ml model. (integration)
Backend: The backend part was the most complex of all. Overall it was written in Flask/Python We used several things and logistics as mentioned below.
- We used Twilio as our platform for handling the webcam feeds. We used Twilio video api for having a real-time video broadcast from multiple participants.
- We created a defined points system. It was linked to the accuracy of the participant’s pose. Points were added and updated every 3 seconds. The participant who had the most points at the end of the session was the winner :D
- We used firebase (firestore) to store data for room sessions (room characteristics and room codes) and points for each user to maintain and update the leaderboard.
- For integrating ML model with the web app and making it work according to our use (detecting the pose accuracy with reference to the given pose) we used the same libraries.
- We used UIUD for room code generation and validation
Challenges we ran into
There were many, many challenges we ran into during this development process.
- Initially, we wanted to get a bunch of poses added to the game, but there were many issues with regards to accuracy. Since our game calculates points based on the accuracy of the pose done, it’s a pretty crucial part and our workaround for this was to downscale the amount of poses to 7 to demonstrate the efficacy of the system and provide an eclectic variety.
- Tailwind’s approach of being component-agnostic made for a “fun” time trying to get the design to look good. It was a low-friction experience designing the home, join, and create room pages - however, creating the room page became quite messy. It took a lot of determination and toying around with the flexbox and grids to solve this,
- Some other issues that arose are included (but not limited to): Some of our internet connections dropping, Heroku rate-limiting us, the CSS grid being dysfunctional, and the music feature being buggy at times
Accomplishments that we're proud of
We are very glad and happy to have a web program that remarkably executes our vision for this app. Many aspects were required for this program to be what it is, from devising a rough plan of what we want to accomplish to training and creating our machine learning model judge. Being able to create a frontend that’s built on the charm and appeal of minimalism and have it seamlessly tied in with a robust backend that is able to link those processes together. Being able to integrate the Twillio API on our application is another awesome accomplishment. Our ML judge was quite accurate thanks to the massive amounts of data and training we poured onto our model.
What we learned
Some of the things we have learned include:
- Communication is key to success.
- Downscaling can be the right approach when time is of the essence.
- You have to be calculated on what you should be persistent with. A minor bug doesn't mean as much as a codebreaking bug.
- Trying to learn Tailwind during a hackathon is a recipe for pain and distress.
- Using ml5js and p5js
- using twilio API for video
What's next for Avastha
All 4 of us are firm believers in the idea that there is room for improvement in anything, and this is the case with our project. The potential for this project is sky high and our roadmap for improvements will include the following:
- Adding more poses and training the model further. We have around 600 images in our current model which handles 7 poses as it stands.
- Allowing for groups of more than 4 people to play and ensuring we don’t face any performance snafus.
- Developing more game modes such as having two teams compete against each other.