Physio Sync

Turn a physiotherapist’s exercise demo into personalised, real-time coaching at home—with OpenCV 5 and on-device AI.

Comment

Inspiration

“Am I doing this correctly?”

For patients practising physiotherapy exercises at home, that question often remains unanswered until their next appointment. A clinician can demonstrate a movement and explain the technique, but a video handout cannot respond when someone loses their position or forgets a cue.

Our team combines physiotherapy, engineering and product experience. We wanted to bring the personalised guidance taught in the clinic into home practice.

That inspired PhysioSync Motion Compiler: turning a clinician’s exercise demonstration into a reviewed, executable coaching program.

What if demonstrating an exercise could also be the way to program its digital coach?

What it does

A physiotherapist records or uploads a demonstration. PhysioSync generates an editable training-card draft and a movement specification containing exercise phases, measured thresholds, coaching cues, quality checks and prepared audio.

The clinician reviews, edits and approves the program before assigning it.

At home, the patient practises in a browser using a single camera. On-device pose estimation provides visual and spoken guidance, counting repetitions and timed holds against the approved movement rules.

The same generic evaluation engine supports different compiled movements, without a separate hand-written evaluator for each exercise.

How we built it

The system connects three components:

  • Clinician workspace: React and TypeScript interfaces for recording demonstrations, reviewing drafts and assigning programs.
  • Motion Compiler: A Python pipeline running on AWS Graviton4, combining OpenCV 5, MediaPipe, Whisper, a vision-language model and deterministic calibration.
  • Patient runtime: Browser-based MediaPipe tracking that keeps the patient’s camera feed on their device.

The compiler prepares the video, masks faces, extracts visual evidence, transcribes instructions and estimates pose. A vision-language model proposes movement phases and rule structures; deterministic calibration measures numerical thresholds from the demonstration. Kokoro generates spoken cues ahead of practice.

The model proposes what to measure, the calibrator measures it, and the clinician decides what to approve.

OpenCV 5 and the compile agent

OpenCV 5 supplies evidence independently of pose estimation: sharpness, subject-region exposure, camera movement, active and still segments, and possible movement outside the frame.

We use optical flow and RANSAC to estimate camera movement, frame differencing to locate motion, and image-quality checks to flag recording problems. OpenCV also supports frame retrieval, enlarged crops, face mosaics and review visualisations.

When the first pass cannot calibrate a usable specification, a bounded agent can inspect this evidence and propose a different phase anchor or rule geometry. Every revision must pass deterministic calibration. The agent cannot supply thresholds or publish programs, and its actions are recorded.

We deployed the CPU worker on AWS EC2 Graviton4 with private S3 storage, Secrets Manager and CloudFormation infrastructure.

Challenges we faced

Finding a measurable reference moment. In a squat demonstration, the first pass selected a rest-phase window with insufficient tracked frames. The agent inspected OpenCV’s still segments and landmark coverage, then selected a standing moment between repetitions. Calibration succeeded, enabling repetition tracking.

Separating subject movement from camera movement. A person filling the frame initially confused our camera-motion detector. Masking the moving subject before tracking background features reduced this false signal.

Making quality checks context-aware. A white wall triggered an exposure warning despite a well-lit subject. We changed exposure analysis to focus on the moving subject where possible.

Preventing invalid repairs. An early agent revision placed rest and target anchors too close together. We added a deterministic separation check and supplied the calibrator’s failure reasons to guide subsequent actions.

Accomplishments

We connected demonstration capture, clinician review and on-device coaching, and ran the compiler and repair agent on AWS.

We also benchmarked stock OpenCV 5 against COOL, the Cloud-Optimized OpenCV Library, on Graviton4. Full-resolution evidence analysis achieved 1.50× speed-up on a plank clip and 1.14× on an arm-raise clip, with matching evidence outputs. Gains were smaller at our current downscaled analysis resolution.

What we learned

Useful coaching requires more than recognising a movement. It needs measurable phases, explicit uncertainty and clinician oversight.

We learned that classical computer vision gives an AI agent concrete evidence for choosing its next action. Profiling also revealed that repeated pose inference dominated several stages, making shared inference a promising optimisation.

What’s next

We plan to expand evaluation, improve HDR handling, reuse pose inference across stages and build durable workflow orchestration.

The repair agent has been exercised on two clips; broader reliability and clinical effectiveness still require validation. Our next step is clinician-guided testing to make home practice clearer and easier to follow.

Built With

Share this project:

Updates

Submission history