SnapDance

Inspiration

We were fascinated by the idea of turning Snapchat into a platform for physical play. Dance games like Just Dance proved that body-movement scoring is universally fun, but they require a console, TV and controllers. We asked, "What if your phone camera was the only hardware you needed?" Snapchat's Lens Studio gave us full-body pose estimation and a turn-based online multiplayer framework, perfect for modernizing this genre and making it more advanced and accessible.

What it does

SnapDance is a Snapchat Lens that uses real-time full-body tracking to score a user's dance performance.

The system:

Tracks 18 skeletal landmarks (nose, ears, shoulders, elbows, wrists, hips, knees, ankles) plus composite points like the hip center, using Snap's ObjectTracking ML pipeline.
Processes uploaded videos to provide real-time references to users
Runs a scoring engine that evaluates pose accuracy at checkpoints, weighted based on how impactful each skeletal landmark is in the current frame of the dance, and normalized to make for consistent and accurate competition
Displays a floating score popup above the user's head that launches at a randomized angle
Wraps the dancer in a body-segmentation aura glow that flashes and pulses in the color representing the quality of their moves for instant feedback
Supports turn-based multiplayer via Snapchat's messaging layer; two friends can take turns dancing, with scores and poses synced for friendly competition

Challenges we ran into

Turn-based data integrity: Snapchat limits data sent between turns, so we had to find a workaround to make sure the same video gets used in every turn. As a solution, we send the metadata of the video between users and compare it to ensure that the same video gets uploaded for fair competition.
Tracking jitter: Raw skeletal positions from the ML model are noisy frame-to-frame. We had to introduce a minimum-frame hysteresis in the state machine so the lens wouldn't flicker between "tracking" and "lost" states during fast movements.
Working with complicated data: Having to adjust and tweak full-body data was difficult; we couldn't use traditional unit tests and so we focused on creating consistent testing environments to test changes.

Accomplishments that we're proud of

Adjusting to new platform: None of us had any prior experience working with Lens Studio, and we're very proud of being able to build an app from just an idea to something published in 24 hours.
Scoring optimizations: We knew we didn't have time to fine-tune scoring weightings for each limb for different body sizes. Instead, we use normalization of limbs through a standardization algorithm and use a velocity-based attention weighting algorithm so that the core parts of dance moves are prioritized more, massively improving accuracy with little fine-tuning necessary.

What we learned

Presentation is everything: a simple idea executed well can work better than more complicated alternatives. Focus on what the end user experiences, and the best way to do that should come naturally
Leveraging existing frameworks is important: For example, creating the 3D tracking model from scratch would be way too ambitious for a hackathon, but knowing how to integrate it into a specialized use case can be just as good.

What's next for SnapDance

Better learning tools: Building alternate modes, like a tutor mode that doesn't score you but rather teaches you at a slower rate, are variations we want to work on
Better UI: This is a barebones version of the UI. In the future, we want to make it more gamified and look professional, less like a tool and more like a fun game that you're encouraged to play with friends
Dedicated backend: We want to add a dedicated backend for a leaderboard and a video database so people don't have to import their own.