Sideline Lift Judge

Film your Olympic lift on your phone and get an AI bar-path score, coaching cues backed by evidence from your own video, and an annotated replay. Built on Nebius + NVIDIA Nemotron.

Comment

Inspiration

I train Olympic weightlifting (snatch, clean & jerk). Bar path is one of the most important things a coach watches: how far the bar drifts away from you decides whether a lift is efficient or gets missed. But most lifters train without a coach watching every rep, and the phone video they film shows them what happened without telling them what was wrong. I wanted a judge on the sideline: film, upload, and get an honest score with evidence.

What it does

Sideline Lift Judge is a mobile-first web app (installable PWA). You film a snatch, clean or jerk on your phone, pick the lift, and upload. You get back:

  • A 1–10 bar-path score and a band (elite to fail)
  • 1–2 coaching cues, each tied to a specific phase of the lift and backed by evidence from your own video: a measurement and a strip of frames, or a clear "seen by the vision model" tag when there's no measurement
  • An annotated replay: the bar path drawn frame by frame, with a dot following the plate, the starting reference line and the furthest-drift point
  • A confidence level and framing warnings. For example, if you filmed at an angle, the app says so and caps confidence instead of pretending to be sure

How we built it

A three-stage pipeline, where each stage does what it's best at:

  1. Measure (OpenCV): a plate tracker fits an ellipse to the plate in each frame, follows the moving plate (not the ones resting on the rack), and measures horizontal drift in plate widths and cm against where the bar was at lift-off. It also detects the lift phases.
  2. Observe (vision model on Nebius Token Factory): openbmb/MiniCPM-V-4_5 looks at key frames and describes what happens in each phase.
  3. Judge (NVIDIA Nemotron on Nebius Token Factory): nvidia/nemotron-3-super-120b-a12b combines the measurements and observations with weightlifting coaching principles, and returns the score, band and cues as structured JSON.

The backend is FastAPI on Heroku. Scoring runs as an async job, and the client polls for the result. The annotated replay is drawn in Python and encoded to H.264 with ffmpeg, so it plays on every phone. Portrait video is treated as the normal case, because that's how people film in the gym.

Challenges we ran into

  • No vision input on Nemotron: the Nemotron models available to us on Token Factory don't accept images. That's why the vision model observes and Nemotron reasons over measurements and text.
  • Tracking the right plate: gyms are full of plates. The tracker first locked onto a plate sitting on the rack, and later onto a far plate in a re-encoded upload. It now prefers the largest, nearest plate that is actually moving.
  • Honest evidence: early versions attached the same evidence to two different cues, or took a "dip" cue's evidence from the lockout. Now each cue's evidence must come from the phase it talks about, or it isn't shown.
  • Camera angle: filming at an angle inflates forward drift. The app now estimates the angle from the plate's ellipse shape, warns the user and lowers confidence.
  • Hosting limits: Heroku's 30-second request limit and a server without system ffmpeg pushed us to async jobs, careful uploads and a bundled encoder.

Accomplishments that we're proud of

  • Every cue shows why: a real measurement from your own video, not a generic tip
  • The app is honest about its limits: confidence and framing warnings are part of the result
  • It works end to end from a phone browser, with no app store

What we learned

  • Combining deterministic measurement (OpenCV) with LLM reasoning gives far more trustworthy feedback than either one alone
  • Treating the LLM as a judge over structured evidence, not as an eye, makes the output easier to verify
  • Language-model scores vary between runs, so showing confidence and evidence matters as much as the number

Limitations

  • Best results come from filming square from the side, with the whole bar and plate in frame; an angled camera can overstate drift
  • Scores can vary from run to run (on our sample clean, earlier runs scored between 4 and 7 with similar drift)
  • Tested mainly on my own lifts so far

What's next

  • More reliable phase detection and calibration in cm
  • History and progress tracking across sessions
  • Native iOS/Android app, and feedback for coaches managing several athletes

Try it: the live demo opens with a pre-recorded sample result (no code needed). Uploading your own clip requires an access code to protect the daily model budget; it's available to judges on request.

Built With

Share this project:

Updates

Submission history