Punch-Out!! AI

Inspiration

Boxing training is expensive and often inaccessible. Personal training sessions can cost upwards of $100/hour, and even group classes are a significant expense over time. Recent advances in VR, AI, and computer vision now make it possible to deliver high-quality, personalized boxing instruction at a fraction of the cost. Punch-Out!! aims to democratize access to expert-level boxing training, allowing anyone to improve their skills safely and effectively, anytime, anywhere.

What it does

Punch-Out!! places you in a virtual boxing gym with your own AI trainer, using a Meta Quest 3. The coach guides you through workouts, gives real-time feedback on your form via computer vision, and motivates you. This initial version concentrates on learning punch combinations. It's akin to having a personal instructor, but readily available. Your movements are tracked, feedback is tailored, and the workout adapts to your progress. It's a personalized, engaging, and convenient way to improve your boxing skills.

How we built it

This project was a fun fusion of different technologies!

• VR Environment: We built the core VR experience in Unity, optimized for the Meta Quest 3. This handles everything from rendering the gym environment to detecting punches and simulating the physics of movement.
• AI Trainer Brain: The backend is built in Python, and it's where the magic happens. We use an OpenAI LLM (o3-mini) to analyze the user's performance (data streamed from the VR headset) and generate appropriate feedback and instructions.
• AI Trainer Voice: We chose Eleven Labs (eleven_flash_v2_5) for our text-to-speech, giving our trainer a distinct and motivating voice.
• Web Interface: A simple React frontend allows users to start workouts. The most important function is the real-time audio streaming, done via WebSockets.
• Server: Handles performance grading, progression, and delivers trainer script text.

The system works as follows: You put on the headset, start a workout, and the AI trainer starts guiding you. Your punches and movements are tracked, the server analyzes them, and the AI trainer gives you feedback and keeps the workout flowing.

Challenges we ran into

• Real-time Audio Latency: Getting the AI-generated voice to the VR headset with minimal delay was crucial for a natural feel. We spent considerable time optimizing our WebSocket implementation to ensure the audio stream was smooth and responsive.
• Computer Vision Accuracy: Precisely tracking punches and form in a VR environment is tricky. We had to experiment with different tracking algorithms and filtering techniques to ensure accurate data was sent to the backend.
• LLM Fine-tuning: Getting the OpenAI LLM to generate consistently helpful and contextually appropriate feedback took some careful prompt engineering and iteration.

Accomplishments that we're proud of

• Seamless Integration: We successfully combined VR, AI, and real-time communication into a cohesive and engaging experience.
• Responsive Feedback: The AI trainer provides surprisingly relevant and timely feedback, making the workout feel dynamic and personalized.
• Proof of Concept: We've demonstrated that it's possible to create a genuinely helpful AI-powered fitness coach using readily available technologies. While we chose boxing as the initial sport, we believe the same concept will work for many other popular forms of physical activity (tennis, golf, dance, etc.)
• Fast Prototyping: We were able to build this in an extremely short period of time, thanks to the newest tools and infra.

What we learned

• Limiting scope to prove out a concept: We initially integrated with Eleven Labs’ conversational AI module, but later realised that since the user didn’t naturally talk much anyway during a workout (i.e. no “taking turns to speak”), we’d not be losing much by sticking to one-way (trainer-to-athlete) for now by using Eleven Labs’ text-to-speech service. This bounded the complexity and allowed us to focus on the core user experience.
• VR Development Nuances: VR development presents unique challenges, especially when dealing with real-time performance and user input.
• Importance of Iteration: Constant testing and refinement were essential to get the feedback loop and user experience right.

What's next for Punch-Out!!

This is just the beginning! We have big plans for Punch-Out!!:

• More Workout Types: We'll add basic punch training, footwork drills, defensive techniques, and even simulated sparring.
• Enhanced Metrics: We plan to provide more detailed performance data, allowing users to track their progress over time.
• Multiple Trainer Personalities: Imagine choosing a drill sergeant, a supportive mentor, or even a celebrity boxer as your AI coach!
• Community Features: Leaderboards, challenges, and the ability to share workouts with friends. We’ll likely take inspiration from Peloton and other social fitness apps on this front.
• We're excited to continue developing Punch-Out!! AI and bring affordable, high-quality fitness coaching to everyone!

Built With

• computer-vision
• elevenlabs
• openai
• python
• react
• typescript
• vr