Fitness Gem

Dual-Stream Biomechanics Coach Powered by Gemini

Comment

Inspiration

We noticed that most fitness apps treat users like data points — tracking reps and calories without truly understanding the individual. For elderly users or those exercising alone at home, there's an added safety concern: what happens if they fall and no one is around? We were inspired by the idea that Gemini 3's massive context window and native multi-modal capabilities could create a coach that genuinely knows you — your injuries, your energy levels, your long-term patterns — and keeps you safe while you train.

What it does

Fitness Gem is a hyper-personalized, multi-modal AI fitness companion that:

  • ** Sees your form via real-time camera pose detection and provides professional-grade corrective feedback after each set using Gemini's video understanding.
  • ** Remembers your entire journey by ingesting continuous "Life Logs" (diet photos, workout history, mood, sleep) into Gemini's 1M+ token context window, detecting long-term patterns like poor sleep correlating with reduced squat depth.
  • ** Keeps you safe with a Guardian Safety System — edge AI detects sudden drops, Gemini verifies whether it's a fall or a rest, and if confirmed, automatically notifies emergency contacts.
  • ** Adapts in real-time — had a stressful day? The AI suggests a restorative flow. Feeling energetic? It ramps up intensity.

How we built it

  • ** Flutter for a cross-platform mobile app (iOS & Android).
  • ** Google Gemini 3 as the central "brain" for reasoning, vision analysis, and long-context coaching.
  • ** Context Caching to store the user's persona (injuries, goals, health history) for instant, low-latency personalized conversations without re-processing massive history each time.
  • ** Google ML Kit (Edge AI) for real-time on-device pose detection with <10ms latency — exercise-specific models handle rep counting and immediate form safety.
  • ** Firebase (Auth, Firestore, Cloud Functions) for the backend, including a Guardian connection system for emergency notifications via FCM.
  • ** A Hybrid Intelligence architecture: Edge AI handles speed-critical tasks (rep counting, fall detection triggers), while Cloud Gemini handles complex reasoning (video form analysis, fall verification, long-term fatigue patterns).

Challenges we ran into

  • ** Nested CoreML model packaging: The .mlpackage was shipped as a zip-within-a-zip, causing silent "Model does not exist" failures on iOS. We had to implement multi-stage extraction with Manifest.json detection to handle both flat and nested bundle structures.
  • ** 60Hz interpolation gap: Camera frames arrive at ~30fps, but our ML model expects 60Hz input. We built a linear interpolation layer to fill the gaps between real frames, significantly improving inference quality.
  • ** Context window management: Balancing the richness of life-log data against token limits required careful caching strategies — deciding what stays cached vs. what gets summarized.
  • ** Fall detection false positives: Distinguishing a user lying down for glute bridges from an actual fall required Gemini's video reasoning rather than simple sensor thresholds.

Accomplishments that we're proud of

  • ** A true hybrid AI pipeline where Edge AI and Cloud Gemini work in concert — local models give instant feedback while Gemini provides deep, contextual analysis.
  • ** The Guardian Safety System with Gemini-powered fall verification — a genuinely life-saving feature for elderly users exercising alone.
  • ** Natural language onboarding that understands your "Why", not just your stats — the AI interview creates a deeply personal coaching profile.
  • ** Achieving professional-grade video-based form analysis using Gemini's native multi-modal capabilities, going far beyond traditional 2D pose heuristics.

What we learned

  • ** Gemini's Context Caching is a game-changer for building AI apps that feel "alive" — users get instant, deeply personal responses without the latency or cost of re-sending their entire history.
  • ** Edge + Cloud hybrid is the right architecture for real-time fitness AI: you can't wait for a cloud round-trip during a squat, but you need cloud intelligence for nuanced decisions.
  • ** Multi-modal AI opens up coaching possibilities that were previously impossible — Gemini can notice things in video that pure landmark-based systems miss entirely (e.g., subtle knee caving, compensatory shoulder movements).

What's next for Fitness Gem

  • ** Real Trainer Integration — Connect with certified personal trainers who can review AI-generated reports, override recommendations, and provide hybrid human + AI coaching. Trainers get a dedicated dashboard with client analytics, session replays, and tools to build custom exercise programs powered by Gemini.
  • ** HealthKit & Health Connect Integration — Sync with Apple HealthKit and Google Health Connect to pull in heart rate, sleep quality, step count, and recovery metrics for a truly holistic coaching experience.
  • ** Expanded Built-in Exercise Library — Ship with 50+ exercises out of the box, each with dedicated ML models, guide videos, and Gemini-powered form analysis — covering yoga, resistance bands, dumbbells, and rehabilitation movements.
  • ** Comprehensive Test Coverage — Build a robust test suite including unit tests for the inference pipeline, widget tests for all UI flows, integration tests for the end-to-end workout session, and automated CI/CD validation to ensure stability as the platform grows.

Built With

Share this project:

Updates