Inspiration
Most people who train alone never get their form checked. Fitness apps tend to do one of two things: plan workouts, or count reps from a camera. They rarely connect the two. We wanted one coaching loop where form checks, workout logs, body metrics and progress photos all feed the same coach.
What it does
AI Fitness Coach watches your form and learns from your training data.
- Log: routines, sets, reps, weight and a rest timer.
- Form Check: record or upload a set. It counts reps and measures range of motion and posture for that exercise. You pick the exercise or let the app detect it, and a Nemotron model turns the numbers into a short plain-language summary.
- Coach: chat answers are grounded in your own sets, body metrics and form results.
- Dashboard: weekly volume, PRs, weight and form trends, BMI and a weekly AI summary.
- Photos: a dated before/after comparison with a cautious AI description. It makes no body-fat estimates.
How we built it
- Pose tracking runs in the browser (MediaPipe), so workout video never leaves the device. Only rep numbers go to the backend.
- Rep counting without hard-coded squat angles. Each rep is a swing of one joint angle between two thresholds, so noise near a threshold can't count twice. For joints $a$, $b$, $c$ with $\vec u = a-b$ and $\vec v = c-b$:
$$\theta = \arccos!\left(\frac{\vec u\cdot\vec v}{\lVert\vec u\rVert\,\lVert\vec v\rVert}\right)$$
We scale $x$ by the video's aspect ratio $w/h$ before computing angles, and smooth with $\tilde\theta_t = 0.6\,\tilde\theta_{t-1} + 0.4\,\theta_t$.
- Automatic exercise detection. The app watches the knee, hip, elbow and shoulder angles and locks onto the joint that swings the most. It sets its thresholds from the person's own range:
$$\text{low} = \theta_{\min} + 0.3(\theta_{\max}-\theta_{\min}),\qquad \text{high} = \theta_{\max} - 0.3(\theta_{\max}-\theta_{\min})$$
Per-exercise rules then flag issues such as shallow depth, elbows drifting or leaning back.
- Backend: FastAPI and SQLite. The coach runs on NVIDIA Nemotron on Nebius Token Factory through its OpenAI-compatible API, and a smaller Nemotron model writes the form and weekly summaries. The frontend is TypeScript.
- Photo comparison: none of the Nemotron models available to us on Nebius accepted images, so the photo analysis uses an image-capable model hosted on Nebius. Photos are only sent when the user presses Analyze.
Challenges
- A 2D camera can't always tell what you're doing. A lateral raise was read as a biceps curl because the arm swings toward the camera and its flat elbow angle jumps. Using 3D pose data is our next fix.
- A hidden squat assumption. Our first version counted every movement as a squat, so we rebuilt the counter around per-exercise joints and removed every squat default.
- Plumbing. API keys, model names and CORS took real debugging, and so did image support, which differs per model.
- Privacy. We kept pose processing on-device and made photo analysis opt-in.
What we learned
A pose model reports where joints are. Deciding what counts as a rep, and whether it was good, is a separate rule-based layer that depends on the exercise and on the camera view. Keeping the LLM away from raw video made the system cheaper, more private and easier to debug.
What's next
3D pose angles that don't depend on camera position, more exercises, a front-view mode, and measured rep-count accuracy on labelled clips. All of our thresholds are starting estimates, tested on synthetic data and our own clips.
Coaching feedback only, not medical advice.
