During quarantine, we all jumped on the trend in downloading TikTok, a social media platform which has taken the world by storm. Known for its short, often dance-related videos, we all jumped in on the fun to learn and create our own TikTok dances. However, what we soon realized was that many of these dances were actually pretty challenging to learn and could take a couple hours to fully master and film. We realized a lot of time was wasted in just overplaying a TikToker's video and comparing it to our own filmed video to spot differences and then refilm. That led us to creating... Dance Pose! 💃🏻
We wanted to put our passion for tech to do something productive for everyone to add another layer to creating these hypnotic dance videos.
What it does
Playing on the words Dance and PoseNet, our website, Dance Pose allows the user to first upload a video of a "professional" doing a solo TikTok dance. It then navigates to the next page where the user can choose to either use a webcam or upload their own filmed video. Dance Pose than takes the two video footages and compares them against each other to ultimately calculate a score (out of 100) as well as return the 3 best/worst timestamps.
This is meant to make dancing to Tiktoks more engaging and offer users a way to gauge their performance and measure improvements!
How we built it
We used TensorflowJS, specifically the PoseNet library. Every pose was represented by 17 keypoints in which we would draw as well as link them to each other to form a skeleton which would represent the dancer's body.
How we compared and scored dance videos
The set of points from each pose are converted into vectors and normalized. We compared two vectors by their cosine distance as well as their Euclidean distance. This gave us a score to be averaged out over many data points. Certain body parts like the arms and elbows were given a heavier weighting since they're also more prominent in most dances.
Challenges we ran into
Many examples of PoseNet usages were using webcams, so one of our biggest challenges was figuring out how to implement PoseNet over video uploads, as it is more geared towards a webcam usage. We also had some troubles with matching the frames as well as calculating an appropriate score to match the similarity in the two videos.
Accomplishments that I'm proud of
As we are newly entering the field of Machine Learning, we are super proud of implementing PoseNet into our application and ultimately launching an interactive website for the user.
What's next for Dance Pose
Currently Dance Pose is only for "single-pose" dance videos. However, we would love to expand it to also work for "multi-pose" dance videos as well! This would mean that friends could come together to see how they perform as a group, and even expand into TikTok dance competitions, where users can compete all over the world to see who can score a higher rating to the original.
As well, the area of motion detection and computer vision is widely applicable to the world of fitness, where we can detect whether our yoga poses or gym moves are up to par. This is only one instance of the rapidly evolving Machine Learning space.