• 

  FitCoachAI app interface on iOS

• 

  FitCoachAI app interface on iOS

• 

  FitCoachAI app interface on iOS

• 

  FitCoachAI app interface on iOS

• 

  FitCoachAI app interface on iOS

Inspiration

My fascination with AI began as it evolved from simple search engines into autonomous systems. Having explored Autonomous Vision in self-driving cars and aerial navigation, I witnessed AI’s ability to recognize speech and faces with precision. However, a piece was missing: a truly interactive, real-time "eye" that could guide us in our physical lives.

The arrival of Gemini Live was the catalyst. It shifted my perspective from building "Static Chatbots" to "Immersive Experiences." I chose the fitness domain because millions struggle with exercise form and injury prevention without professional guidance. I wanted to create a coach that lives in your pocket but watches you with the expertise of a professional trainer.

What it does

FitCoachAI is an elite, high-precision athletic trainer powered by Gemini 2.5 Flash. It transforms an iPhone into a multimodal sensory hub that:

• Sees: Monitors your environment and 19+ skeletal joints in real-time.
• Analyzes: Uses biomechanical logic to detect form errors (like "knee collapse" or "back rounding").
• Talks: Provides instant, sub-second native audio feedback and manages sets, reps, and rest intervals.
• Adapts: Automatically detects and speaks the user’s native language while maintaining professional fitness terminology.

How we built it

We engineered a "Multimodal Symphony" by integrating a high-performance iOS frontend with a Google Cloud backend:

1. The Eyes (Frontend): Built with Swift and the Vision Framework, the iOS app extracts normalized skeletal coordinates.
2. The Brain (Backend): A FastAPI orchestrator running on GKE Autopilot uses the Google AI Edge SDK to feed compressed spatial data to Gemini.
3. The Logic: We implemented a vertical displacement and angular stability monitoring system.
4. The rep counting logic is defined by:

$$ \text{Rep Count} = \begin{cases} +1, & \text{if } y_{hp} \text{ moves from } y_{start} \to y_{max_depth} \to y_{start} \ 0, & \text{otherwise} \end{cases} $$

Challenges we ran into

• **Latency vs. Precision: **Streaming raw video and 19 skeletal joints at high frequencies created massive data overhead. We solved this by building a Coordinate Compression Engine, converting raw numbers into "spatial shorthand" (e.g., hp(0.5,0.8)), which reduced token consumption by 60%.
• Multilingual Fluency: Ensuring the agent could switch languages (e.g., from "Two more!" to "Ещё два!") without losing its "persona" or technical accuracy required deep prompt engineering and leveraging Gemini’s native multilingual capabilities.
• Environment Noise: Handling connections where the user moves out of frame required the agent to recognize "Spatial Calibration" issues and give commands like "Step back, I can't see your feet." ## Accomplishments that we're proud of
• Sub-second Response Time: Achieving near-human timing for audio corrections (e.g., yelling "Knees out!" exactly when the error occurs).
• Zero-Ops Scaling: Successfully deploying a complex multimodal backend on GKE Autopilot that handles elastic loads automatically.
• Biomechanical Awareness: Creating an agent that doesn't just "chat" but actually understands the physics of a squat or a deadlift.

What we learned

Building FitCoachAI taught us that the future of AI isn't in massive raw data, but in intelligent data fusion. We discovered that while video provides context, skeletal coordinates provide the truth of movement. Fusing these inputs allows Gemini to understand human intent versus actual physical execution. We also gained deep experience in real-time streaming using Secure WebSockets and the Google AI Edge SDK.

What's next for FitCoachAI

• Wearable & Biometric Integration: Connecting Apple Watch to stream real-time heart rate data. This will allow the agent to dynamically adjust rest intervals based on the user's actual cardiovascular recovery (HRV) rather than a fixed timer.
• Knowledge Retrieval with RAG: Implementing a Retrieval-Augmented Generation (RAG) pipeline connected to an extensive database of gym equipment and specialized training complexes. This will enable the agent to provide expert-level technical tutorials for any machine the user encounters.
• Comprehensive Nutrition Intelligence: Expanding the agent’s capabilities to track and analyze daily, weekly, and monthly intake of Macronutrients (Proteins, Fats, Carbohydrates) and total calories.
• Goal-Oriented AI Dietitian: Providing personalized nutrition plans tailored to specific user objectives:
  • Weight Loss (Caloric Deficit management).
  • Muscle Gain (Hypertrophy-focused Caloric Surpluses).
  • Maintenance (Neutral energy balance for peak athletic form).
• Multi-User "Group Class" Mode: Developing computer vision logic to monitor and motivate multiple people in a single frame, enabling interactive group sessions.
• Long-term Memory & Progression: Utilizing persistent storage to help FitCoachAI remember past performances, tracking personal records (PRs) and ensuring a truly personalized, long-term athletic progression.

Built With

• adk
• audio
• fastapi
• gemini
• gke
• ios
• skeleton
• swift
• video
• vision
• websockets