Landing Page
Sign Up
Sign In or Login
Account Manager
Dashboard
Dashboard Achievements Insights
Dashboard Progress Insights
Quick Coach Assistance
Workout Dashboard
Real-Time Tracking
FitCoach one-on-one Session
Leaderboard for Community
Tutorials Page

🤖 FitTrackAI: Webcam-First Fitness Platform with AI Pose Detection & Gamification

💭 Inspiration

Most fitness apps die for the same reason: they ask you to log every workout by hand. That feels like homework, and homework loses to Netflix. I wanted to build something where the friction of "tracking" disappears entirely — your camera does the work, and the app rewards you for showing up.

The other half of the inspiration was Duolingo. A green streak and a leaderboard rank turned a billion people into language learners. There's no reason fitness can't borrow the same trick — except we get to layer real‑time computer vision on top of it.

🎯 What It Does

FitTrackAI is a webcam‑first fitness platform with three big ideas:

🎥 AI Pose Detection – Ten exercises tracked in real time: squats, lunges, glute bridges, push‑ups, bicep curls, shoulder press, lateral raises, tricep extensions, sit‑ups, and crunches. The browser counts your reps, gauges joint angles, and gives live form cues like "go a bit lower" or "locked out — nice."
🎮 Real Gamification – Every rep earns XP. PRs trigger confetti. Eighteen achievements unlock as you progress (Bronze → Silver → Gold → Platinum). A daily streak counter pushes consistency, and a tier‑based leaderboard ranks athletes from Bronze to Champion.
🧠 An AI Coach That Actually Knows You – FitCoach (streaming Llama 3.3 on Groq) has your full history in its system prompt: streak, level, total reps, recent workouts, PRs, achievements. Ask "plan my next workout" and it references the squats you did yesterday.

Plus: an activity heatmap (GitHub‑style, 13 weeks), a personal‑records dashboard, a tutorials section with embedded videos, and a landing page with a live demo.

🏗️ How We Built It

Layer	Technologies
Frontend	Next.js 14 (App Router, server actions), React 18, TypeScript
Auth	Clerk (middleware‑gated routes)
Database	Neon Postgres + Drizzle ORM
Pose Detection	TensorFlow.js + MoveNet (Lightning), WebGL backend @ ~30 fps
AI Coach	Groq + Llama 3.3 70B, Vercel AI SDK
Styling	Tailwind CSS, shadcn/ui, Radix, canvas‑confetti
Deployment	Vercel

Key Architecture

The pose detector is a "use client" component that loads TF.js dynamically (ssr: false). Each exercise has a config:

{
  direction: "flex" | "extend",
  restThreshold: number,
  peakThreshold: number,
  measure: (keypoints) => number | null
}

The rep counter is a simple state machine: rest → peak → rest = 1 rep.

Server actions handle the rest: write to Neon, detect PRs, recompute aggregates, evaluate 18 achievements, and update XP using the curve:

$$ \text{XP needed for next level} = 100 \times \text{current level} \times 1.5 $$

The Coach API injects live user data into every request and streams responses via streamText + toUIMessageStreamResponse.

⚡ Challenges We Ran Into

Challenge	Solution
Noisy keypoints (±5° jitter → flickering counter)	EMA smoothing ((\alpha = 0.35)), 3‑frame confirmation, 120 ms peak hold, 400 ms refractory window
Sit‑ups/crunches wrong geometry	Re‑derived thresholds from real bent‑knee angles + added camera hints (`📷 camera at floor level, side‑on`)
Drizzle correlated subquery bug	Switched to `LEFT JOIN ... GROUP BY users.id` instead of nested subqueries
Clerk v7 requires Next 15	Pinned `@clerk/nextjs@^6` for Next 14 compatibility
1,700‑line HTML pose detector	Rebuilt as a typed React component with a generic `flex/extend` config

LaTeX for the EMA smoother

$$ \hat{\theta}t = \alpha \cdot \theta_t + (1 - \alpha) \cdot \hat{\theta}{t-1}, \quad \alpha = 0.35 $$

🏆 Accomplishments We're Proud Of

✅ Real‑time browser ML at 30 fps – no upload, no GPU, privacy‑preserving (video never leaves your machine)
✅ Grounded AI coach – knows your streak, PRs, and skipped days; advice quality is night and day
✅ Unified rep engine – one flex/extend primitive handles 10 different exercises
✅ Polish everywhere – confetti on PRs, glowing streak flame, tiered leaderboard with podium, floating coach widget, bento landing page with live demo

📚 What We Learned

"Hysteresis isn't enough — you also need temporal stability." Confirmation frames + a refractory window kill transient spikes.

Server actions + revalidatePath beat REST APIs for this scale. One line updates the dashboard after a workout.
Grounded LLMs > smart LLMs – a smaller model with the user's actual history outperforms a giant model with no context.
Drizzle's type inference is a superpower – but subqueries and raw sql need explicit aliasing.
Bento grids work when the math is right – the fix was a 4+2 split, not creativity.

🔮 What's Next for FitTrackAI

Short‑term (next 3 months)

[ ] More exercises – plank holds (timer‑based), pull‑ups, mountain climbers, jumping jacks. Need a third "hold" mode.
[ ] Sound + voice – TTS rep counts ("eight… nine… ten…") and optional voice mode for the Coach.
[ ] Daily quests & weekly bosses – randomized 3‑quest daily set + community boss ("50,000 squats this week").

Long‑term vision

[ ] Coach‑driven workout sessions – the Coach prescribes a full session with rest timers.
[ ] Friends + clans – friend invites, friends‑only leaderboard, group challenges.
[ ] Mobile PWA – offline support and "add to home screen" for the gym.
[ ] Skill tree – unlock pistol squats, diamond push‑ups via per‑exercise milestones.