deepdance
video link Improve your dancing skills
What it does
David and Caroline Dancing To A Choreograph. David scored 0.337 (from 0 to 1, higher is better).
DeepDance is an app that allows you to upload a video of your favorite dancer that you're trying to emulate and records you copying their choreography. Our algorithm compares your movements with the original dancer and gives you a score describing how well you matched them and gives you useful feedback to improve! After the analysis, it also lets you playback a "skeleton" of yourself next to the original dancer so that it's easier to spot key differences.
Our inspiration
Now that everyone's stuck in quarantine (hopefully), you probably have nothing better to do than post Tik-Toks every day. But if you're like us, your dancing probably isn't as good as you want to be. Wouldn't it be awesome if you could use data-science to improve your dance skills? That's exactly what DeepDance does.
How we built it
We built this application in Python and Javascript using:
Challenges we faced
- Robust and fast pose estimation
- Video to video synchronization
- Pose comparison and evaluation
- Front-end to back-end communication
OpenPose
Our app uses a pose estimation deep learning model called OpenPose to determine the position of important keypoints on the dancers' bodies in the camera frame. OpenPose synthesizes confidence maps and Part Affinity Fields produced by a convolutional neural network. Together, they give enough information to accurately and quickly label or predict where various body parts are located. At each frame in the videos, we compare key metrics such as joint angles and velocities (10 joints total) between the two dancers to determine the difference. Finally, we process the difference between the key metrics to generate a score for each joint and an overall score, as well as feedback for what areas the dancer should focus on to improve.
Scoring metric
Determining how well a dancer is replicating another is not a trivial task. We spent some time thinking about what differentiates two dancers. We settled on defining a distance metric between the joint angles of the two dancers at each frame to determine the "error". After some research, we found researchers who were tackling a similar problem of comparing two dancers. They took a very similar approach and selected a Mahalanobis distance metric between joint angles. In our case, with the data being a single video source, the Mahalanobis distance metric simplifies to a Euclidean distance. For each joint, we then average this error metric over the whole video length to get a error estimate.
David and Caroline Dancing To YMCA. David scored 0.9336 (from 0 to 1, higher is better).
Using this metric, we were able to see differences in scores for videos where we tried to emulate a dancer versus ones where we danced randomly. However, the error estimate for each joint is in units of radians, which isn't very useful for the user. Additionally, each joint has a different range of values. For example, the elbow joint can range from nearly 0-180 degrees, whereas the hip may only move from 150-180 degrees. This makes the weight of each joint different in the final score.
To solve this we calculated a z-score of the error. We selected a standard deviation for the z-score a function of the range of possible values for each joint angle (which we empirically found through a bunch of videos of us dancing). By making the standard deviation a function of the range, we are effectively scaling all of the scores to be on a similar range. By adjusting the actual function, we can change how much we spread out the data (which is useful to make data understandable by the user, it's more pleasing to see a difference of 55 -> 80 vs. 0.55 -> 0.57). Using the Z-score, we calculate the percentile and then scale the data to the right range. Finally, we find the average of the individual joint scores to calculate the overall score, which is on a scale of 0-100.
We are pretty happy with this metric as it is pretty straight-forward, but does a good job of identifying differences between dancers. We saw clear performance differences between videos where we tried to dance well versus in videos where we didn't.
Syncing Videos
One of the challenges we didn't initially anticipate was that the two videos we're comparing may not be synced. In other words, the two dancers may start dancing at different timesteps. However, we are lucky with this project as both videos will have the same song playing in the background, to which the dancers will be synced. Therefore, all we need to do is match the audio on the two videos to sync the dancing. We did a little research and found a repo on Github that had done exactly that. The syncing function performs a fast Fourier transform on the audio to get the signature in the frequency domain. It then performs pattern matching on frequency constellations to determine the offset required to sync the videos.
The app was last modified in 2014 and was written in Python 2, so we had to make a couple of changes to run in Python3 and work with the latest packages. We plan to clean them up and submit a PR to the repo so that other people can continue to use the package.
Demonstration of the UI.
The Result!
Accomplishments we're proud of
We like the simple, yet well-performing metric we created to score the dancer. We're also infatuated with the skeletons of the dancers we generate because they are such a useful way to see how you should moving (they also look hilarious).
What's next for deepdance
- Interactive scores that are highlighted during the video
- Evaluative feedback of scores
- Improving our metric with temporal tolerance
- Creating a recommended instructors database
- Expand our app into mobile devices
Acknowledgements
- Caroline Janssen for being our super helpful dance instructor!
Built With
- caffe
- electron
- html
- javascript
- opencv
- openpose
- python
Log in or sign up for Devpost to join the conversation.