Inspiration
The COVID-19 pandemic has made it extremely difficult to pursue many hobbies, one of them being dance. Conventional dance classes are out of the question, and one-way online tutorials do not provide an engaging experience. We wanted to create a digital platform that uses AI to make learning dance fun and interactive - all from the comfort of your home!
What it does
Tango is an AI Dance Coach that allows dancers to practice and receive live feedback for any dance that they choose. Our product utilizes MediaPipe’s Pose Estimation library in conjunction with OpenCV to detect dancers’ positions and compare them with the dance that they wish to learn. We make use of dynamic time warping (DTW) and vector calculations to sync the dancer’s video with their desired original dance, in both the time and space dimensions. Our live annotations allow users to track where their poses should be in relation to their limb lengths and torso position. After recording a dance rehearsal, users are able to play it back and analyze specific portions of their routine using our live charting and accuracy metrics.
How we built it
- Python, FastAPI and OpenCV for the back-end
- Dynamic time warping to account for time delays
- Angular vector calculation to account for size disparity
- ReactJs + ChakraUI + Mediapipe + Recharts for the front-end
- FFMPEG for mp4 encoding and track merging
- SQL Lite + Python for leaderboard feature
Challenges we ran into
- Getting mediapipe to work in the front-end, along with React
- Getting the ideal position annotations working in sync with the actual position.
- Interpolating dynamic time warping scores into a percentage scale
- Working with MP4 files and form data
- Syncing the audio track from the coach video with the video track in from OpenCV
Accomplishments that we're proud of
- Getting the annotations and data visualization components working really well.
- Creating an clean and comprehensive front-end interface
- Getting everything done in < 24 hours!
What we learned
- Working with mediapipe and Reactjs
- Building complex user interfaces in ChakraUI
- Using Recharts for data visualization
- Using FFMPEG with Python
- Working with form data in FastAPI
What's next for Tango
We are super excited to take Tango forward as an open source project. A core feature we would like to add is multiplayer functionality for dancing with your friends. We would also like to make our user interface more responsive to various screen sizes. Lastly, we would like to ensure robust error handling for edge cases.
Log in or sign up for Devpost to join the conversation.