Formalates

Inspiration

The idea for FORMALATES sits at the intersection of two things I care about: Pilates as a discipline of precise, deliberate movement, and biotech's obsession with turning the body into readable data. Pilates is anatomically exact — a few degrees of spinal curl or a millimeter of shoulder tension separates good form from injury — yet it's still taught almost entirely by eye, with no objective feedback loop for people practicing without a live instructor.

The Sports track's nod to athlete transitions (LeBron leaving the Lakers) got me thinking about how elite athletes get monitored constantly — HRV, load, biomechanics — while everyday Pilates practitioners get none of that. I wanted to close that gap: what if your webcam could read your form the way a lab reads vitals?

What it does

FORMALATES turns a laptop webcam into a live Pilates form coach, starting with The Hundred. It:

• Tracks 33 body landmarks in real time using MediaPipe Pose, entirely in-browser
• Computes a torso curl angle at the shoulder joint to flag whether you're too flat or overcurled
• Monitors shoulder-to-ear distance (normalized by torso length) to catch shrugging/neck strain
• Counts pump reps via wrist oscillation relative to the hip
• Displays a running form score, a live sparkline of the curl angle over the last 12 seconds, and a scrolling feedback log — styled like a vitals monitor rather than a fitness app

No footage leaves the device, and no account or setup is required beyond granting camera access.

How we built it

The core computation is angle geometry applied to pose landmarks. For a joint $b$ with adjacent landmarks $a$ and $c$, letting $\vec{v_1} = a - b$ and $\vec{v_2} = c - b$, the joint angle is $\theta = \arccos\left(\frac{\vec{v_1} \cdot \vec{v_2}}{|\vec{v_1}||\vec{v_2}|}\right)$. For torso curl, $a$ is the ear, $b$ is the shoulder, and $c$ is the hip — so $\theta$ shrinks as the spine curls up off the mat and widens as it flattens.

Stack and architecture:

1. MediaPipe Pose (@mediapipe/pose) runs pose inference client-side via WASM; drawing_utils renders the skeleton overlay on an HTML canvas, mirrored to feel like a mirror rather than a recording.
2. Vite + React, structured so logic and presentation never mix: a single usePoseTracking hook owns all MediaPipe setup, the detection loop, and scoring math. Components are purely presentational — they just render props.
3. Tunable thresholds (curl angle bounds, shrug ratio, rep-detection sensitivity) live in one isolated constants.js file, so they can be adjusted quickly during rehearsal without touching logic.
4. Design direction: dark graphite background, teal "readout" accent, monospace numerics, and a live sparkline — visually closer to an ECG monitor than a workout app, to reinforce the biosignal framing.

I grounded the approach in two peer-reviewed sources rather than guesswork:

• Li, D. & Chen, C. (2022). Research on exercise fatigue estimation method of Pilates rehabilitation based on ECG and sEMG feature fusion. BMC Medical Informatics and Decision Making, 22, 67. (https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-022-01808-7) Twenty subjects performed 150 minutes of Pilates rehabilitation while ECG and sEMG signals were recorded simultaneously and classified into fatigue states. This confirmed that real biosignal research treats multi-signal fusion as the standard for reliable state detection — a useful benchmark for where FORMALATES could grow beyond pose alone.

• Lee, K. (2021). The Relationship of Trunk Muscle Activation and Core Stability: A Biomechanical Analysis of Pilates-Based Stabilization Exercise. International Journal of Environmental Research and Public Health, 18(23), 12804. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8657451/) This study compared trunk muscle EMG activity between experienced and inexperienced Pilates practitioners alongside 3D motion-capture kinematic data. It directly validated the assumption FORMA relies on: that kinematic/pose data meaningfully correlates with Pilates movement quality, even without EMG hardware.

Challenges we ran into

• Translating biomechanics into thresholds. The cited research measures muscle activation directly via EMG electrodes; FORMALATES only has joint coordinates from a webcam. Approximating "good form" using geometry alone meant careful, iterative threshold-tuning rather than clinically-derived numbers — and being transparent in the UI that these are heuristics, not medical guidance.
• Rep-counting robustness. Wrist-oscillation-based rep detection is sensitive to how large someone's arm pumps are; small movements risked being missed entirely, requiring a balance between sensitivity and false positives.
• Time pressure. With a matter of hours left at build time, scope had to shrink fast: one exercise instead of the multi-exercise system originally envisioned, and no hardware biosignal integration (HRV, EMG) despite the research pointing that direction.

Accomplishments that we're proud of

• Shipped a fully working, real-time CV pipeline with zero backend — inference, scoring, and rendering all happen client-side, keeping latency low and ensuring no user video ever leaves the browser.
• Grounded a hackathon heuristic in published, peer-reviewed EMG/kinematic research, and were upfront in the UI about where the heuristic ends and validated science begins.
• Built a distinct visual identity — a biosignal "vitals monitor" aesthetic — that reinforces the concept instead of just skinning a generic fitness UI.

What we learned

• Kinematic (pose-only) tracking is a legitimate, research-backed proxy for movement quality, but true biosignal fidelity (EMG, ECG) captures things geometry alone can't — muscle fatigue, activation timing, effort.
• Client-side ML has real product advantages beyond convenience: privacy and latency are genuine features here, not just consequences of not having time to build a backend.

What's next for Formalates

• Multi-exercise support — Roll-Up, Single Leg Stretch, and Teaser, each with its own joint-angle logic
• Session history — the first feature that would actually require a backend and lightweight auth, to track form-score trends over multiple sessions
• Biosignal integration — pairing pose data with heart-rate or HRV input (phone-camera rPPG or a paired wearable), moving closer to the fused-signal approach used in the Li & Chen (2022) fatigue-classification study
• Cycle-aware and adaptive coaching — adjusting intensity/feedback sensitivity based on user-reported context, extending the feminist-health-tech angle behind the original concept
• Exportable session reports — a simple PDF/summary judges or physical therapists could actually use post-session

Built With

• biomechanics
• biosignal-processing
• client-side-ml
• computer-vision
• fitness-tech
• healthtech
• javascript
• mediapipe
• mediapipe-pose
• no-backend
• npm
• pose-estimation
• react
• react-hooks
• sports-tech
• vite
• web-app
• webrtc