gymlens

Workout interface
Demo interface

Inspiration

Most of us train alone. Personal trainers are expensive, and a mirror cannot show every part of your movement while you exercise. We wanted to recreate what a good training partner provides: someone who follows each rep, gives timely feedback, and helps you understand your progress—without expensive equipment or uploading personal workout videos.

What it does

GymLens is a browser-based workout and movement coach. Place your phone or laptop where your body is visible, build a workout, and start moving:

Live movement tracking — tracks 33 body landmarks and draws a skeleton over the camera feed
Automatic rep and set counting — recognizes completed movements using exercise-specific joint angles and movement phases
Supported technique cues — identifies measurable patterns such as limited range, torso movement, elbow drift, and alignment changes
Voice coaching — delivers rep counts and practical cues using ElevenLabs, with a browser-voice fallback
Guided camera setup — shows framing requirements, calibration progress, and an automatic start countdown
Video analysis — analyzes a prerecorded workout clip locally in the browser
10 supported exercises — squat, push-up, bicep curl, lunge, Romanian deadlift, overhead press, glute bridge, bent-over row, tricep dips, and pull-ups
Multi-exercise workouts — lets athletes choose exercises, sets, reps, weight, rest time, and a workout name
Session reports — records sets, reps, timing, joint angles, supported cues, and available arm data
Workout history and scheduling — stores completed workouts and upcoming plans
Progress charts — graphs recorded weight over time with exercise-specific filtering
AI commentary — Gemini explains recorded results and can compare them with previous sessions
Private pose processing — camera and uploaded-video pose tracking run on the device; raw video is not sent to the GymLens backend

The muscle diagram is driven by tracked joint movement. It is a visual movement estimate, not a direct measurement of muscle activation.

How we built it

The frontend uses React, TypeScript, and Vite. MediaPipe Pose Landmarker runs in the browser using WebAssembly and available browser acceleration, keeping pose inference on the user’s device.

For each frame, GymLens calculates joint angles from three body landmarks. For joint (B), between points (A) and (C):

$$ \theta = \arccos\left( \frac{\vec{BA}\cdot\vec{BC}} {\lVert\vec{BA}\rVert \lVert\vec{BC}\rVert} \right) $$

Each exercise has its own movement analyzer. It processes smoothed joint angles through phases such as ready → eccentric → concentric. Timing, range, calibration, and stability checks help determine whether a completed movement should count as a rep.

Bicep curls track both arms independently and combine completed left-and-right cycles into one workout rep, whether the arms move together or alternate.

Supported movement rules attach cues to recorded reps. A feedback engine prioritizes and rate-limits spoken guidance so it does not repeat the same instruction continuously.

The backend uses FastAPI and SQLAlchemy, with SQLite for local development and PostgreSQL with TimescaleDB through Tiger Data in production. Tiger Data stores athlete profiles, scheduled workouts, workout groups, exercises, sets, reps, weights, timing, measurements, cues, and saved AI commentary.

Auth0 protects athlete accounts. ElevenLabs generates a consistent coaching voice, with audio cached to reduce cost and delay. Gemini receives structured workout measurements and recent-session context to generate grounded post-workout commentary.

The frontend is deployed on Vercel, and the API runs on Render.

Challenges we ran into

Unstable landmarks — raw pose landmarks can shake, disappear, or briefly switch sides. We added smoothing, visibility requirements, stable calibration, and movement-phase checks.
Counting completed reps — partial movements, fast gestures, pauses, and unrelated movements could resemble exercise reps. Each exercise required separate timing and range rules.
Tracking both arms — curls needed independent left-and-right state machines so simultaneous and alternating curls behaved consistently.
Camera placement — some exercises work best from the side while others need a front view. GymLens provides exercise-specific setup guidance and checks whether required joints are visible.
Useful feedback without overclaiming — a camera cannot evaluate every part of someone’s technique. Reports distinguish supported cues from unmeasured aspects and avoid presenting the result as a universal form score.
Consistent mobile coaching — users may place their phone several feet away. We redesigned setup and live workout screens so framing, countdowns, reps, and tracking status remain visible without scrolling.
Reliable voice output — mobile browsers restrict audio playback. We added an explicit voice-unlock step, phrase caching, provider backoff, and browser speech fallback.
Controlling API usage — coaching uses bounded phrases and cached audio, while Gemini runs after completed sessions instead of on every camera frame.
Shipping quickly as a team — parallel work across pose tracking, authentication, storage, voice, reports, and deployment required careful integration.

Accomplishments that we’re proud of

We built a workout coach that:

Runs pose estimation locally in a browser
Counts reps and sets across 10 exercises
Supports multi-exercise workout plans
Gives spoken feedback during a set
Works with a live camera or prerecorded video
Stores structured training history securely
Turns raw joint measurements into understandable reports
Shows exercise-specific weight progression over time
Works as a practical mobile web experience without requiring a native application or wearable

What we learned

We learned that pose estimation is only the first step. Lighting, clothing, camera placement, landmark visibility, and differences between bodies all influence the signal.

The difficult problem was converting noisy frame-by-frame measurements into stable events people could trust. Calibration, smoothing, movement phases, timing requirements, and clear recovery behavior mattered as much as the pose model itself.

We also learned that AI coaching becomes more useful when it is grounded in structured evidence. Gemini receives measured reps, timing, angles, cues, load, and previous-session context rather than raw video or an unrestricted prompt.

Finally, we learned how to ship a full-stack application with authentication, real-time computer vision, voice generation, AI commentary, persistent time-series training data, and two production deployment platforms under hackathon constraints.

What’s next for GymLens

Our next step is bringing GymLens to iOS and Android as a native mobile application. A dedicated mobile experience will make camera positioning, workout setup, voice coaching, and live rep tracking faster and more user-friendly.

Native mobile support will also let us improve performance, provide clearer camera and audio permissions, save workouts for offline use, send workout reminders, and create an experience designed specifically for a phone placed several feet away during training.

Future plans also include: