Vision of Project
Handling and interpreting user movements is a difficult tasks for computers. Using pose estimation we can map the human anatomy to select vertices and save this data, frame by frame. Training a data model against 100+ hours of video of people moving around allows us to have near perfect estimates of the action being performed on screen.
After watching one of my friends open up her tiktok, and started trying to replicate the movments she saw on the video, i noticed that there were alot of people who desired to be as sleek as the people in those videos. Without realizing, there was a community of people who desired to achieve the dance moves seen in the video's causing us to build MIMIC.
What it does && How we built it
Given a video where the user is dancing and achieving complex moves, our system will take there skeletal movements mapping them to specific angles that there legs, arms and torso achieved at different frames in time. These important frames are then used to test the user's ability to copy those movement vectors, where they attempt to mimic the original videos moves.
We looked around for a way to estimate poses, and came across a technique called heatmapping with convolutional neural networks. Unfortunately, the data to run the models was too large, and we had to resort to a pretrained model using this technique.
Challenges we ran into && What we learned
Throughout the project we were challenged to read abstract papers on effective ways of reading and parsing skeletal structures of humans and apply the correct model/classifiers to tensorflow.
Accomplishments that we're proud of
Never before have we had a project that we envisioned so clearly at the beginning, building out the domain model along with subsequent view design.
What's next for MIMIC
Building out the gallery space for other users to see the pass rates of other submissions. We want to create a culture where players are excited to challenge each other to create videos that are harder and harder to mimic. In the future we plan on further improving the platform, displaying at our local school's highlight even ImageRIT.