I've been going to physiotherapy for three years become of a knee injury that ended my career in martial arts. It started with knee pain and escalated from there. Just because it happened to me, doesn't mean it has to happen to everyone else. Modifying tools developed at Stanford and Google, we've created a full physical suite that captures properties about your skeletal and muscular system from just a video. After that, we run rigorous simulation testing to reverse-engineer what's causing your joint or muscle pain, and provide physiotherapy suggestions.

How we built it

We split the work across two platforms running in parallel, synchronized at two integration points:

Mac side: MediaPipe pose extraction on real footage, a parametric motion generator, and a RAG + LLM layer that turns the biomechanical output into a plain-language explanation and corrective plan, plus the demo frontend with skeleton overlays and a convergence readout. Windows side: OpenSim validation (scaling the musculoskeletal model to the subject, then running inverse kinematics and inverse dynamics), a Bayesian optimization loop (Optuna with its GP-based sampler) that searches for corrected motion parameters minimizing lumbar joint moment, and a scoring/diff stage that compares baseline vs. corrected mechanics.

Challenges we ran into

OpenSim's environment constraints. OpenSim only runs cleanly in its own conda environment, separate from everything else in the pipeline (the optimization loop, LLM tooling), so every call into it had to go through a subprocess bridge rather than a direct import, which shaped a lot of the architecture. Our first instinct was to score form against "safe" joint angle ranges, but the biomechanics literature is unclear on what a safe squat angle even is, flexion during a squat is normal, not inherently dangerous. We ended up scoring on net lumbar joint moment instead (already available from our inverse dynamics output, and the same underlying quantity established ergonomics thresholds are built on), which was both more defensible and cheaper to compute. Every candidate evaluation costs real time (an OpenSim inverse dynamics run), so we had to size the optimization budget, trial count, GP startup trials, around what a real recording session could actually afford, rather than assuming unlimited compute. Cross-platform coordination. Splitting Mac and Windows work meant the interface between them (parameter schemas, log formats, output shapes) had to be nailed down before either side could build against it with confidence — a few late changes on one side rippled into the other.

Moving on from struggles, accomplishments that we're proud of!

Getting a full pipeline: pose extraction, real OpenSim biomechanical simulation, Bayesian optimization, and natural-language explanation, working end to end across two machines and two runtimes, with a scoring approach grounded in actual biomechanics research rather than a plausible-looking heuristic.

What we learned

How to translate biomechanics research (joint moments, ergonomics thresholds) into a concrete, computable optimization target.

The practical realities of Bayesian optimization when each evaluation is expensive: budget planning, startup trial tuning, and persistence matter as much as the choice of sampler.

What's next

Extending the joint set we track beyond the current sagittal-plane markers, tightening the loop between real inverse-dynamics output and the optimizer's score, and eventually moving from a pre-recorded demo to a live, real-time correction loop.