FitAI

FitAI is an AI-powered fitness platform that combines multi-agent AI systems with modern web technologies to provide personalized fitness coaching, nutrition guidance, and real-time workout tracking.

Comment

HackSeneca (FitAI) - Project Story

💡 Inspiration

The inspiration for HackSeneca came from witnessing the struggle many people face when trying to maintain consistent fitness routines, especially without proper guidance. During the pandemic, we observed how traditional gym experiences and personal training became inaccessible, leaving people to navigate their fitness journeys alone.

We were particularly inspired by:

  • The democratization of AI: How advanced AI could make personalized fitness coaching accessible to everyone
  • Computer vision breakthroughs: MediaPipe's real-time pose detection opening new possibilities for form analysis
  • Multi-agent systems: The potential of specialized AI agents working together like a real fitness team
  • Voice analysis research: Studies showing how vocal patterns can indicate physical fatigue and wellness

Our vision was simple yet ambitious: What if everyone could have a team of AI experts - a fitness coach, nutritionist, and wellness analyzer - available 24/7?

🎯 What it does

HackSeneca (FitAI) is a comprehensive AI-powered fitness platform that provides:

🤖 Multi-Agent AI Coaching System

  • Manager Agent: Intelligently routes user queries to appropriate specialists
  • Fitness Coach: Creates personalized workout plans, analyzes exercise form, and provides training guidance
  • Nutritionist: Develops meal plans, analyzes food images, and offers dietary advice with visual meal generation

📱 Real-Time Exercise Analysis

  • 33-point pose detection tracking body movements at 30+ FPS
  • Automatic rep counting with phase detection for exercises like push-ups, squats, planks, lunges, and burpees
  • Form accuracy scoring providing real-time biomechanical feedback
  • Visual pose overlay with live skeleton visualization

🎤 Advanced Fatigue Detection

Using our custom VGG19 + Ensemble Learning model: $$\text{Fatigue Score} = f(VGG_{features}(S_{mel})) \circ PCA_{reduction} \circ Ensemble_{classifier}$$

Where:

  • $S_{mel}$ represents 196×196 mel-spectrograms from 50-second audio clips
  • $VGG_{features}$ extracts deep CNN features using pre-trained VGG19
  • $PCA_{reduction}$ compresses features for optimal performance
  • $Ensemble_{classifier}$ provides robust binary fatigue classification

🍎 Intelligent Nutrition Tracking

  • AI-powered food image recognition and nutritional analysis
  • Automated meal planning with visual meal generation using Azure FLUX.1-Kontext-pro
  • Real-time nutrition insights and macro tracking

🛠️ How we built it

Architecture Design

We implemented a hierarchical multi-agent architecture using CrewAI, ensuring intelligent task delegation:

graph TD
    A[User Input] --> B[Manager Agent]
    B --> C[Fitness Coach Agent]
    B --> D[Nutritionist Agent]
    C --> E[Workout Plans & Form Analysis]
    D --> F[Meal Plans & Food Analysis]

Backend Development

  • FastAPI: Chose for its async capabilities and automatic API documentation
  • CrewAI Framework: Implemented specialized agents with distinct roles and tools
  • Azure OpenAI Integration: Primary LLM for intelligent conversational AI
  • Custom Tool Development: Built FluxImageGenerator for meal visualization

Frontend Implementation

  • Next.js 14: Leveraged App Router for modern React development
  • MediaPipe Integration: Implemented real-time pose detection in the browser
  • Framer Motion: Added smooth animations and transitions
  • Shadcn/ui + Tailwind: Created a modern, accessible component system

Machine Learning Pipeline

Our voice fatigue detection system processes audio through multiple stages:

  1. Audio Preprocessing: Convert to 8kHz, 50-second clips
  2. Spectrogram Generation: Create 196×196 log-mel spectrograms using: $$S(m,n) = \log(|STFT(x)[m,n]|^2)$$
  3. Feature Extraction: Use pre-trained VGG19 to extract deep features
  4. Dimensionality Reduction: Apply PCA for feature compression
  5. Ensemble Classification: Multiple classifiers for robust prediction

Real-Time Pose Analysis

Implemented custom algorithms for exercise-specific analysis:

  • Push-ups: Arm angle and body alignment tracking
  • Squats: Hip and knee angle analysis using joint coordinates
  • Planks: Core stability and body line detection
  • Rep Counting: Phase detection with minimum hold time validation

🚧 Challenges we ran into

1. Multi-Agent Coordination

Challenge: Preventing agents from overstepping their domains Solution: Implemented strict delegation rules in the Manager Agent with clear routing logic:

# Delegation Rules:
- Workout/Exercise queries → Fitness Coach
- Nutrition/Food queries → Nutritionist  
- Ambiguous queries → Clarification request

2. Real-Time Pose Detection Performance

Challenge: Achieving 30+ FPS pose detection while maintaining accuracy Solution:

  • Optimized MediaPipe pipeline for browser performance
  • Implemented efficient Canvas rendering for pose overlay
  • Used web workers for heavy computational tasks

3. Voice Fatigue Model Accuracy

Challenge: Balancing model complexity with real-time performance Mathematical Approach: $$\text{Optimization} = \arg\min_{\theta} \left[ \mathcal{L}(y, f_{\theta}(x)) + \lambda \cdot \text{Complexity}(\theta) \right]$$

Solution:

  • Used pre-trained VGG19 to reduce training time
  • Applied PCA for dimensionality reduction without significant information loss
  • Implemented ensemble methods for improved robustness

4. Integration Complexity

Challenge: Coordinating multiple AI services (Azure OpenAI, Azure FLUX, MediaPipe) Solution:

  • Created modular service integration with fallback mechanisms
  • Implemented comprehensive error handling and retry logic
  • Used environment variable configuration for flexible deployment

5. Real-Time Data Synchronization

Challenge: Keeping pose detection, rep counting, and form analysis synchronized Solution: Implemented event-driven architecture with React hooks for state management

🏆 Accomplishments that we're proud of

🔬 Technical Innovations

  1. Hierarchical AI Agent System: Successfully implemented domain-specific AI agents that work collaboratively
  2. Real-Time Exercise Analysis: Achieved accurate pose detection and rep counting at 30+ FPS
  3. Advanced Fatigue Detection: Developed a sophisticated ML pipeline using VGG19 + Ensemble Learning
  4. Seamless Integration: Combined multiple AI services into a cohesive user experience

📊 Performance Metrics

  • Pose Detection: 33-point body tracking at 30+ FPS
  • Rep Counting Accuracy: >90% accuracy across 5 exercise types
  • Fatigue Detection: Binary classification with ensemble reliability
  • Response Time: <2 seconds for AI agent responses

🎨 User Experience

  • Intuitive Interface: Modern, responsive design with smooth animations
  • Real-Time Feedback: Immediate visual and textual feedback during workouts
  • Personalization: Context-aware recommendations based on user data
  • Accessibility: ARIA-compliant components for inclusive design

🏗️ Architecture Excellence

  • Scalable Backend: FastAPI with async processing capabilities
  • Modular Frontend: Reusable components with TypeScript type safety
  • Robust Error Handling: Comprehensive exception management and fallbacks
  • Documentation: Extensive technical documentation for maintainability

📚 What we learned

Technical Learnings

  1. Multi-Agent AI Systems: Understanding how to design agents with clear boundaries and effective communication protocols
  2. Real-Time Computer Vision: Optimizing MediaPipe for browser-based pose detection
  3. Audio Signal Processing: Converting audio signals to meaningful ML features using spectrograms
  4. Ensemble Learning: Combining multiple models for improved prediction reliability

Development Insights

  1. API Design: Creating intuitive, well-documented APIs with FastAPI
  2. State Management: Managing complex state in React applications with custom hooks
  3. Performance Optimization: Balancing functionality with real-time performance requirements
  4. Integration Patterns: Coordinating multiple external services reliably

Project Management

  1. Team Collaboration: Effective division of responsibilities across our 4-member team:

    • Lead AI Engineer: Multi-agent architecture and CrewAI implementation
    • Full-Stack Developer: Frontend and API integration
    • Backend Engineer: FastAPI and AI model integration
    • ML Engineer: Computer vision and voice analysis features

  2. Iterative Development: Importance of rapid prototyping and user feedback incorporation

Mathematical Concepts

Applied advanced mathematical concepts in practical implementations:

  • Linear Algebra: PCA for dimensionality reduction
  • Signal Processing: Fourier transforms for spectrogram generation
  • Probability Theory: Ensemble methods for classification
  • Optimization: Model parameter tuning and performance optimization

🚀 What's next for FitAI

Short-Term Goals (3-6 months)

  1. Enhanced Exercise Library: Expand to 15+ exercise types with specialized form analysis
  2. Social Features: Community challenges, progress sharing, and peer motivation
  3. Wearable Integration: Connect with fitness trackers for comprehensive health monitoring
  4. Mobile Application: Native iOS/Android apps for on-the-go fitness tracking

Medium-Term Vision (6-12 months)

  1. Advanced Biometric Analysis:

    • Heart rate variability analysis
    • Sleep quality integration
    • Stress level monitoring through voice analysis

  2. Personalized Adaptation: $$\text{Adaptation Rate} = \alpha \cdot \frac{\partial \text{Performance}}{\partial \text{Time}} + \beta \cdot \text{User Feedback}$$

  3. Injury Prevention: Predictive models for injury risk based on movement patterns

  4. Nutrition Optimization: Meal timing and macro distribution optimization

Long-Term Aspirations (1+ years)

  1. Virtual Reality Integration: Immersive workout experiences with VR environments
  2. Clinical Applications: Partnerships with healthcare providers for rehabilitation programs
  3. Global Expansion: Multi-language support and cultural dietary preferences
  4. Research Contributions: Publishing findings on AI-assisted fitness and voice-based health monitoring

Technical Roadmap

  1. Model Improvements:

    • Implement transformer-based models for better context understanding
    • Develop federated learning for privacy-preserving model updates
    • Advanced computer vision for detailed biomechanical analysis

  2. Infrastructure Scaling:

    • Kubernetes deployment for auto-scaling
    • Redis caching for improved performance
    • PostgreSQL database for production data management

  3. AI Enhancements:

    • Multi-modal learning combining vision, audio, and text
    • Reinforcement learning for personalized workout progression
    • Edge computing for offline functionality

Research Opportunities

  1. Voice Biomarkers: Exploring vocal patterns for broader health insights
  2. Movement Quality Assessment: Developing metrics for movement efficiency
  3. Behavioral Pattern Analysis: Understanding user motivation and engagement patterns

HackSeneca represents our commitment to democratizing fitness through AI innovation. By combining cutting-edge technology with practical application, we're building a platform that doesn't just track fitness—it understands, adapts, and evolves with each user's unique journey toward better health.

Built With

Share this project:

Updates