ShuttleCoachAI

ShuttleCoach uses Gemini 3's multimodal vision to analyze action sequences (Prep-Hit-Finish) for kinetic flaws and powers an interactive AI Coach to provide tailored drills via session context.

Comment

Inspiration

Badminton is a sport of precision. Many enthusiasts record their training but struggle to identify subtle technical flaws without professional guidance. Professional analysis tools are often prohibitively expensive or complex. Our vision was to leverage Gemini 3’s reasoning and multimodal capabilities to transform any training video into objective, actionable coaching advice.

What it does

  • Automatic Action Analysis: Intelligent recognition of core actions like Smashes, Clears, and Lifts.
  • Biomechanical Insights: Visualizes 10+ metrics (e.g., impact height, coordination) via radar charts.
  • Multimodal AI Diagnosis: Gemini 3 "watches" the action sequence to pinpoint technical issues.
  • Interactive AI Coach: Real-time Q&A based on analysis results for deeper tactical understanding.
  • Social & Growth: Personalized share cards and performance trend tracking.

How we built it

Built with a decoupled frontend/backend architecture:

  • Pose Engine: MediaPipe Pose extracts temporal data from 33 body keypoints.
  • Analysis Layer: A Python backend pre-calculates biomechanical metrics (e.g., torso rotation).
  • Gemini 3 Core: The soul of the project. We feed metrics and keyframe sequences (Prep -> Hit -> Follow-through) into Gemini 3, enabling it to "understand" the kinetic flow and provide human-like, deep diagnostic feedback.
  • UX Implementation: A React-based frontend providing a seamless experience for analysis, sharing, and interaction.

Challenges we ran into

  • Action Classification (Biggest Challenge): Differentiating similar motions (e.g., Clear vs. Smash) in non-controlled environments with varying camera angles was difficult. We implemented a hybrid solution: heuristic rules for initial screening, followed by Gemini 3’s multimodal reasoning to verify the action type by interpreting the visual context, effectively overcoming high-angle or occluded-view errors.
  • Latency vs. Accuracy: Optimizing API calling strategies to ensure high-quality multimodal analysis while maintaining a fast response time for a smooth user experience.
  • The Art of Feedback: Balancing professional coaching rigor with encouraging language suitable for beginners.

Accomplishments that we're proud of

We are most proud of achieving "Visual-Metric Synergy." Unlike traditional tools that only show dry numbers, ShuttleCoach enables the AI to "eye-witness" the posture and provide specific advice like "Your elbow didn't fully extend during the hit." This bridge between raw data and professional knowledge demonstrates the true potential of AI in athletic education.

What we learned

  • The Power of Multimodality: We discovered that LLMs like Gemini 3 can solve CV edge cases through visual reasoning.
  • Human-Centric AI: We learned to balance cold metrics with warm coaching language, making the AI's output truly valuable and actionable.

What's next for ShuttleCoachAI

  • 3D Reconstruction: Transitioning from 2D pose to more accurate 3D trajectory analysis.
  • Perspective-Specific Scoring: Applying different weights for side-view (better for prep analysis) and rear-view (better for directionality).
  • Match Analysis: Moving from single-action analysis to full-match tactical execution and footwork tracking.
  • Social Ecosystem: Introducing community rankings to boost engagement within the badminton community.
Share this project:

Updates