Have you ever had a time where you want to learn how to dance, perhaps for an upcoming performance or ball, but dance lessons are way too expensive? This application provides a personal assistant that gives you feedback based on your dance moves!

What it does

This application provides a personal assistant that assigns an aggregate score based on how well you match the poses from a professional dancer of that particular dance. First, you record yourself in the web application. Then, the Siamese Neural Network will calculate and find a dance from an existing collection of Youtube videos that closely matches your moves. You may also opt to go random by recording an arbitrary 1 second video, which allows the neural network to act in a random manner as there is still some work in refining the machine learning algorithm to predict this accurately. The aggregate score is calculated by weighting the scores of the OpenPose frames from 17 hinges based on a human's position. The score is subsequently updated in the web application, along with a recap of your performance with the OpenPose's matplotlib pose indicators to see how you compare to the professional dancer.

How we built it

On the frontend, we used React, HTML, CSS, and JavaScript to develop the web application. The web application uses the react-multimedia-capture npm library to record the poses of the user. Then, it stores the video as a blob, and the link is posted to the ngrok server.

On the backend, the ngrok server does processing and uses the Flask server to employ the OpenPose computations of the user's recorded video. Also, we used the Siamese Neural Network to predict what dance video closely resembles your dance movements from the collection of Youtube videos and uses OpenPose to determine the wireframes of the body. Then, these two videos are compared and assigned a weight score which subsequently increments the user's score in the web application.

Challenges we ran into

It was a bit difficult to set up OpenPose and extracting the user wireframes. File transfers are rather difficult and the ngrok server remedies the issue to an extent. Latency in the OpenPose calculations due to heavy computation power reduced the frames per second to single digits, which made it a bit difficult to accurately calculate the user's score on the web application.

What's next for InsDance

We would like to optimize the existing OpenPose calculations of the body wireframes to increase the frames per second of the output video. To enhance user experience, we would like to add a larger variety of dances to fulfill more users' preferences, which can be done accurately as we continue to improve the neural network model.

Built With

Share this project: