Inspiration
There are a lot of dance trends on social media today. This means that many people all over the world who feel inspired by a dance can learn how to do it the same way by understanding what moves they are doing wrong and how they can go about fixing it.
What it does
The user inputs a reference video of a dance they would like to do, and then uploads a video of themselves doing the same dance. Then, our model calculates all moving joints from the reference video and pins them on the users video. Analyzing each frame of the video, and where the current landmarks are, it then gives a score based on how well the move was executed. Once all frames have been analyzed, a total score out of 100 will be given to the user, as well as the frame and timestamp that they could improve upon.
How we built it
We built this project by first finding a pretrained model on human joint detection, as training a model of this size would require a lot of time. For this, we went with MediaPipe's Pose Landmarker. Once we had the model, then we built upon it to analyze videos and break down each frame of the video and having an overlay of all of the landmarks (or joints). We then used
Challenges we ran into
Some issues we faced with this project was firstly creating the scoring system based on how well each move was hit. This meant that both videos had to be processed and then each frame looked at individually and having a certain range of error in which the joint had to be without any penalty. Additionally, we had to make sure that the same video would give us a high score as well.
Accomplishments that we're proud of
We’re proud of building a full-stack system that brings together real-time video processing, AI pose estimation, and user-friendly feedback. We managed to take a relatively complex task of comparing dynamic body movements and turn it into something intuitive and engaging. The scoring system and visual overlays were also key highlights that we were excited to get working smoothly.
What we learned
We learned how to work with a pretrained model for pose estimation and more importantly how to integrate into a custom pipeline for our project. And then linking the back-end to the front-end was quite an interesting process because that's how the whole thing came together.
What's next for Must Dance!
We plan to further work on the accuracy of the movements and will try to provide more helpful feedback instead of just timestamps in which the move was wrong. For example, if the user didn't quite do a step properly, we can provide verbal feedback on how to fix it the next time.
Log in or sign up for Devpost to join the conversation.