Dance Flow AI

Transform any video into a dance challenge! Real-time pose tracking with MediaPipe, instant scoring, and Gemini 3 AI coach providing personalized feedback to help you master every move.

Comment

Inspiration

Dance Flow AI was born from the idea of making dance learning accessible, fun, and interactive. We wanted to combine the excitement of rhythm games like Dance Dance Revolution with the power of AI to create a personalized dance coaching experience. The vision was to allow anyone to learn any dance routine from any video, with real-time feedback and AI-powered coaching.

What We Learned

Building this project taught us several valuable lessons:

  • Real-time Pose Detection: We learned how to leverage MediaPipe Pose for accurate, low-latency human pose estimation in the browser
  • Vector Mathematics: Implementing cosine similarity algorithms to compare user poses with reference poses in real-time
  • AI Integration: Integrating Gemini 3 API to generate contextual, personalized feedback based on performance metrics
  • Performance Optimization: Managing multiple video streams, canvas rendering, and pose processing at 60fps without lag
  • Time Synchronization: Developing robust time-matching algorithms with tolerance offsets to handle video playback variations

How We Built It

The project follows a modular architecture:

  1. Video Analysis Phase: The Analyzer component uses MediaPipe Pose to extract pose landmarks from the reference video at 0.1-second intervals, storing them in a time-indexed Map structure.

  2. Real-time Game Engine: The GameEngine component captures user movements via webcam, processes them through MediaPipe, and compares them against reference poses using cosine similarity calculations.

  3. Scoring Algorithm: We developed a weighted scoring system that evaluates 10 key body limbs (arms 40%, legs 40%, torso 20%) using vector mathematics. The algorithm calculates similarity scores ranging from -1 (opposite direction) to 1 (perfect alignment).

  4. AI Feedback Integration: After each session, Gemini 3 Flash generates personalized coaching feedback based on accuracy percentage, max combo, and session duration, providing encouraging and constructive guidance.

  5. User Experience: Built with React and TypeScript for type safety, Tailwind CSS for modern UI, and canvas-confetti for celebratory effects when users achieve high scores.

Challenges We Faced

  • Time Synchronization: Matching user movements with reference video timestamps required implementing tolerance offsets (±0.2 seconds) to handle natural timing variations
  • Performance: Processing two video streams (reference + webcam) simultaneously while maintaining 60fps rendering was challenging. We optimized using requestAnimationFrame and efficient canvas operations
  • Pose Matching Accuracy: Fine-tuning the cosine similarity algorithm to accurately penalize wrong movements (negative scores) while rewarding correct poses
  • Browser Compatibility: Ensuring MediaPipe works consistently across different browsers and handling CORS issues for video URLs
  • Real-time Feedback: Implementing a live suggestion system that analyzes the worst-performing body part every 3 seconds without impacting performance

The result is a fully functional, real-time dance game that makes learning dance moves engaging and accessible to everyone!

Built With

Share this project:

Updates