FormCoach AI

Every athlete deserves a coach. FormCoach AI turns any webcam into a real-time form coach that counts your reps, scores every rep out of 100, and speaks corrections out loud — 100% in the browser, with zero video ever leaving your device.

💡 Inspiration (the problem)

Most gym injuries come from bad form — improper technique is behind roughly half of them, and bad squat form alone causes a large share of lower-back strains in new lifters. A real coach fixes your form, but a real coach costs more per hour than many students spend on food in a week. So people turn to fitness apps — and the apps want money every month, stop working without internet, and worst of all, send your camera video to their servers.

With the World Cup on and millions inspired to train right now, the gap isn't motivation — it's feedback. I wanted a coach that is private, free, offline, hands-free — and one that remembers you. So I built it, solo, with nothing but the webcam everyone already owns.

🏋️ What it does

FormCoach AI watches you train and coaches you like a human coach would:

• Five drills, every rep measured — squats, push-ups, bicep curls, shoulder press, and jumping jacks. Each drill has its own joint-angle state machine over 33 tracked body landmarks and its own fault vocabulary.
• Scores every rep 0–100 — squat depth, torso lean, knee position, body line, elbow drift — with a live form-score ring. (A coaching aid, not a medical device.)
• Speaks corrections out loud ("Go deeper", "Chest up") prioritized by injury risk, so you never look at a screen mid-set.
• Fully hands-free — say "squats", "start", "finish" and it obeys; ask it anything and it answers out loud. Designed for complete eyes-free operation, so blind and low-vision athletes can train independently.
• Every mistake is photographed. When a fault fires, that exact frame — skeleton overlay included — is captured into a timestamped evidence gallery. The photos go to a vision AI running on my own device, which looks at each image and writes two things: what the problem is, and how to fix it.
• Download the session as a book. One click builds a page-turning flip-book — cover, every fault photo with the AI's written diagnosis and fix, the timestamped error notebook, the coach's verdict — generated as one self-contained file on the athlete's device.
• A Movement Twin. Other trainers compare you to fixed thresholds. FormCoach captures your own best rep as a private baseline, replays it as a ghost skeleton, and measures every rep against it: "Depth 11° short of your best." When fatigue breaks your form, it names the exact rep where the decline started.
• The coach remembers you. It knows your name, your sessions, your scores, your repeated mistakes, your records, your injuries, and your goal — through a self-hosted Memobase memory server that turns every session and chat into a long-term athlete profile. All on your machine.
• 4 parallel AI agents write your coaching report — Biomechanics, Injury Risk, Programming, Progress — each returning a score, findings, and visible reasoning. Transparent AI, not a black box.
• A team dashboard for a human coach — token-gated: each athlete trains privately in their own browser; the coach sees form trends and injury-risk flags for the squad. Athletes need no accounts, so there is no user database to breach.
• Zero-knowledge accounts + privacy you can watch. Training history is AES-256-GCM encrypted with a key derived from the user's password (the password never leaves the device), and a live on-screen counter shows frames analyzed on-device vs. video bytes uploaded — always zero, verifiable in DevTools.

🛠 How I built it (tools & architecture)

The whole system runs on one laptop — zero cloud, zero API keys, zero cost:

• In the browser (video never leaves): Camera → MediaPipe Pose (33 landmarks @ 30 fps, WebAssembly + GPU) → a biomechanics engine I wrote from scratch — vector math for joint angles, per-exercise finite-state machines for rep detection, and a fault-deduction scoring model. It feeds the voice coach, the Movement Twin, the flip-book, and the encrypted history vault. Vanilla JS, no framework, no build step — installable as a PWA.
• The privacy bridge: only JSON numbers — angles, scores, faults — ever leave the browser. Never video.
• Backend: FastAPI (Python) runs the four AI agents concurrently with asyncio.gather, each constrained to Pydantic-validated structured output, persisted in SQLite.
• Local AI: Ollama — Llama 3.1 8B powers the agents and coach chat, Llama 3.2 extracts memories, and moondream (a small vision model) actually looks at the fault photos.
• Long-term memory: self-hosted Memobase (Docker: API + Postgres + Redis) distills sessions and conversations into each athlete's profile — recurring faults, corrected faults, PRs, injuries, goals — which flows back into every answer the coach gives.
• Voice both directions: Web Speech API — synthesis with priority levels and cooldowns so the coach talks like a human, not an alarm; recognition with a command grammar and barge-in.

♟️ Challenges I ran into

• Rep detection that doesn't double-count. Raw joint angles are noisy; I solved it with hysteresis — different enter/exit thresholds per phase — in each drill's state machine.
• Chrome vs. local AI — the hardest bug of the weekend. The hosted app suddenly couldn't reach Ollama while localhost worked perfectly. Root cause: Chrome's new Local Network Access rule silently blocks public HTTPS pages from loopback addresses. Fix: every AI request now declares targetAddressSpace: "loopback", turning a silent block into a one-click permission prompt, plus an in-app banner that explains it.
• A coach that finishes its sentences. Naive text-to-speech cancels the current utterance for every new line, so the coach kept cutting itself off mid-word. Now routine lines queue behind the current sentence, and only critical safety cues interrupt.
• Coaching cadence. My first version screamed every fault every frame — a fire alarm with opinions. Cues are now prioritized (critical > warning > info) and throttled, and a held bad position counts as one fault, not hundreds.
• Converting pixels to centimetres for the jump test with a single camera — solved by calibrating against the athlete's own body: average standing nose-to-ankle length over 30 frames, scaled by the user's real height.
• Fallbacks that tell the truth. A mistyped AI endpoint used to kill the chat silently. Now the app probes the endpoint, falls back to auto-detected local Ollama, and an engine tag under every reply names the model that actually answered.

🏆 Accomplishments I'm proud of

• A complete, working, deployed product — built solo, in the hackathon window.
• Real biomechanics, not an API wrapper: every angle threshold is based on published coaching standards.
• Privacy by design — the architecture itself, not a policy page, is what protects the athlete.

🧠 What I learned

State machines beat ML classifiers for rep counting when you have good landmarks; browser GPU inference is now fast enough for real sports use; voice UX needs as much design as visual UX; and a local vision model on a laptop can genuinely look at a photo of your squat and coach you.

🤞 What's next

Sport-specific drills (cricket bowling action, football agility, basketball shooting form), side-by-side comparison with a pro athlete's reference movement, and team mode pilots for school sports programs.

Built With

• asyncio
• canvas
• css3
• docker
• fastapi
• html5
• javascript
• llama
• mediapipe
• memobase
• moondream
• ollama
• postgresql
• pwa
• pydantic
• python
• redis
• sqlite
• svg
• web-speech-api
• webassembly