VibeScan

Analyze dance movements in real-time with AI

Comment

Inspiration

The global phenomenon of K-Pop and viral dance challenges on platforms like TikTok has made dancing more popular than ever. Everyone wants to join the fun, but learning and perfecting moves can be difficult and intimidating. Private lessons are expensive, and group classes don't offer personalized feedback. We noticed a gap: while many people practice by watching YouTube videos, there's no easy way to get immediate, objective feedback on your movements.

We were inspired by the idea of a personal AI dance coach—a tool that's accessible, private, and fun. We wanted to empower anyone, from a casual K-Pop fan to a dedicated dance trainee, to improve their skills from the comfort of their own room. Our goal was to capture the essence of a "vibe"—that hard-to-define quality of a great dance performance—and translate it into understandable data, making VibeScan the perfect project for a hackathon celebrating "vibe coders."

What it does

VibeScan is a web-based platform that acts as your real-time AI dance coach. Using just your device's webcam, VibeScan analyzes your body movements as you dance. Here’s a breakdown of its core functionality:

Real-time Pose Tracking: It uses advanced vision AI to map 33 key points on your body, creating a live skeleton that mirrors your every move.

Performance Analysis: VibeScan doesn't just watch; it evaluates. It scores your dance based on six key metrics: Power, Speed, Gaze, Balance, Rhythm, and Expressiveness.

Instant Feedback: You see your scores update live on screen through intuitive gauges and graphs, allowing you to make adjustments on the fly.

YouTube Integration: You can search for any song on YouTube and have it play directly within the app, so you can practice your favorite choreography with the actual music.

Privacy-Focused: All analysis happens directly in your browser. Your video feed is never uploaded or stored on a server, ensuring your practice sessions remain completely private.

How we built it

We built VibeScan using a modern, efficient, and privacy-conscious tech stack.

Frontend Framework: We chose React 18 with TypeScript for a robust, scalable, and type-safe user interface. The entire application is a Single Page Application (SPA) built using Vite for a blazing-fast development experience.

Styling: Tailwind CSS was used for rapid, utility-first styling, allowing us to create a clean, responsive, and mobile-first design. Icons are provided by Lucide React.

Core AI Engine: The heart of VibeScan is Google's MediaPipe Tasks Vision library. We use its PoseLandmarker model to detect 33 body landmarks in real-time from the webcam feed. This runs entirely on the client-side using WebGL and WASM acceleration, which is crucial for both performance and user privacy.

External APIs: We integrated the YouTube Data API v3 to search for music and the YouTube IFrame Player API to embed and control the video player, synchronizing the music with our analysis loop.

Deployment: The project is deployed on Netlify, connected directly to our GitHub repository for seamless continuous integration and deployment (CI/CD).

Challenges we ran into

Translating Data into "Vibe": The biggest challenge was converting raw coordinate data from MediaPipe (e.g., the position of a wrist or an ankle) into meaningful metrics. Defining an algorithm for "Rhythm" or "Expressiveness" was a process of trial and error. We had to experiment with calculating velocity, acceleration, movement symmetry, and periodicity to create scores that genuinely reflected the quality of the dance.

Performance Optimization: Running a sophisticated AI model in a browser in real-time is demanding. We initially faced low frame rates on less powerful devices. We overcame this by implementing adaptive frame skipping, optimizing our rendering logic on the HTML5 Canvas, and leveraging Web Workers to offload some processing from the main UI thread.

Synchronization: Perfectly syncing the YouTube audio with our real-time video analysis was tricky due to unpredictable buffering and API latencies. We had to build a resilient timing engine that could handle these inconsistencies to ensure our "Rhythm" score was accurate.

Accomplishments that we're proud of

We are incredibly proud of creating an application that feels both magical and genuinely useful. Seeing your own skeleton move in perfect sync with you on screen for the first time is a "wow" moment. We successfully managed to build a complex, real-time AI application that runs entirely in the browser, making it accessible to anyone without installation and with full privacy. Most importantly, we've built a tool that makes learning to dance less intimidating and much more fun.

What we learned

This project was a deep dive into the world of browser-based machine learning. We learned how powerful and accessible libraries like MediaPipe have become, enabling developers to build AI-powered experiences without needing a backend or extensive ML infrastructure. We also learned invaluable lessons in performance optimization, algorithmic design, and the importance of user-centric design when translating complex data into a simple and engaging user experience.

What's next for VibeScan

VibeScan is just getting started. We have a clear roadmap for the future:

Comparison Mode: Allow users to upload a reference video (e.g., a professional dancer) and see a side-by-side comparison with their own performance, including a real-time similarity score.

User Profiles & Progress Tracking: Implement user accounts to let dancers track their scores over time, save their favorite practice sessions, and see their improvement.

Community & Social Features: Introduce dance challenges, leaderboards, and the ability to share results with friends or a coach.

Advanced Analytics: Provide more detailed post-session reports, including heatmaps of movement and analysis of specific body parts to pinpoint areas for improvement.

Built With

Share this project:

Updates