Dancem8

Inspiration

The COVID-19 pandemic has presented the world with all sorts of challenges—big and small—and in our experience, pursuing our passions virtually hasn’t been a lot of fun. Inspired by personal struggles with virtual dance classes, we decided to make an AI-based dance assistant so that as long as you’re enthusiastic about dance, the pandemic doesn’t hinder you from mastering your art.

What it does

Dancem8 allows the user to upload an image of their attempt at the dance step they wish to hone. It then runs a pose estimation analysis on the user's uploaded image and compares the pose to that of a professional dancer through a self-written algorithm. The comparison returns qualitative feedback that helps the user understand what they're doing wrong and how to correct it.

How we built it

Coffee in, code out! Even though that pretty much describes the whole process but here's some technical jargon: Once the user uploads their image of a step, Dancem8 transforms the user's image to normalize it. It then converts the transformed image object into a tensor and uses Tensorflow's implementation of the PoseNet model for executing 2D pose estimation on the image. The pose estimation process returns the keypoints of body parts in the user's image. To take into account the different body shapes and sizes, Dancem8 uses a custom algorithm that calculates the distance of the user's different body parts from a common point on the 2D skeleton. After calculating all the distances in the user's image, Dancem8 compares the distances to that in a professional dancer's image. Keeping in mind that, realistically, dance steps can not be replicated exactly, it compares the distances after removing a certain threshold value from the distances. Finally, the difference in the distances is converted into qualitative feedback through a custom dictionary which is then shown to the user for improvement.

Challenges we ran into

Tensorflow's Pose Estimation model was not at all easy to understand. It required a lot of hard work and effort to understand how it works and how to implement it in our pipeline. It's also our first time doing this math-intensive project so it was definitely very difficult to accomplish. Generating data from professional dancers' images was also slightly hard because the exact same poses were not easily available.

Accomplishments that we're proud of

The proudest thing for us would most certainly be successfully implementing Tensorflow's Pose Estimation model along with all the mathematical functions to compute the key points in the images.

What we learned

We learned a lot about human pose estimation and its various types. Since we tried both single-pose estimation and multi-pose estimation, it was a really enriching learning experience.

What's next for Dancem8

We plan to add the functionality to allow the users to add new moves to the Dancem8 by simply adding a professional's images of the steps. We also want to make this a more convenient process for the users by adding a feature of uploading their videos directly from which we'll extract frames and pass it on to our pipeline.

Built With

• express.js
• javascript
• json
• node.js
• posenet
• tensorflow
• yarn