GIF

Inspiration

Motion capture is how games, films and sports science get believable human movement, but a marker-based studio costs more than USD 150,000, needs markers glued to the body and a trained operator (Uhlrich et al., 2023). Indie game developers, animators, physiotherapists and coaches mostly go without. Single-camera AI apps are cheap, but they guess depth from one view, so the motion drifts and shakes – unusable for animation and unreliable for measurement. Everyone already owns a phone, so we asked: can three to five ordinary, unsynchronized phones replace the studio?

What it does

AIMoCap turns video from 3–5 phones into a calibrated, smooth 3D skeleton with 133 whole-body keypoints (body, feet, hands, face) and exports it as BVH for Blender, Unreal Engine and Unity. It runs end to end on a CPU cloud server on AWS, with OpenCV 5 doing the vision work.

Find the cameras – each phone films a checkerboard; OpenCV 5 recovers where every phone stands.
Find the person – three neural networks on the OpenCV 5 DNN engine locate 133 keypoints per view.
Line up the clocks – the phones were never started together; AIMoCap finds each phone's delay from the motion itself.
Build the 3D skeleton – robust multi-view triangulation plus a smoother that removes jitter without slowing fast movement.
Export – BVH, previewed on a game character.

A judge can open the web app, browse full results, or press Run live: three phones × 2 s go from raw video to 3D on two CPU cores in about 6.4 minutes, with every stage shown as it happens.

How we built it

OpenCV 5 is the engine, not just a dependency:

Calibration: findChessboardCorners on a 2× upscaled frame + cornerSubPix, findChessboardCornersSB as fallback, camera pose from solvePnPGeneric (IPPE). The board was found in 90 / 90 phone clips (SB detector alone: 66 / 90). Worst error against the lab's own calibration: 0.19° and 15.4 mm.
Inference: YOLOX person detector, RTMW-x whole-body pose and RTMPose hand model run on cv2.dnn. Keypoints match ONNX Runtime to ≤ 0.005 px. On CPU, OpenCV 5 is faster, which is what lets the demo run on a GPU-less cloud server.
Geometry: undistortPoints, projectPoints and Rodrigues power synchronization, triangulation and every reprojection check.

CPU inference speed, OpenCV 5 vs ONNX Runtime:

Model	OpenCV 5 (`cv2.dnn`)	ONNX Runtime
RTMW-x, per crop	378 ms (−27 %)	520 ms
RTMPose hand model	73 ms	145 ms
YOLOX person detector	30 ms	97 ms
End to end, per camera frame	0.39 s	0.45 s

Our own algorithms on top:

Geometric synchronization: an epipolar (Sampson) search initialises pairwise offsets; a joint multi-camera refinement minimises reprojection error on frames where both cameras clearly see the person; a parabola gives sub-frame precision. No clapper, audio or hardware sync.
Triangulation: soft per-camera trust instead of hard inlier switching, then a bidirectional constant-acceleration Kalman/RTS smoother with Huber weights and speed-dependent process noise, so fast motion keeps its speed. The noise is defined in continuous time, so one setting works at 30 and 60 fps.
2D stability: temporally smoothed person crops, sub-pixel SimCC decoding, calibrated keypoint confidence, dedicated hand and face crops.

AWS (deployed by one script, no console clicks): CloudFront (HTTPS) → EC2 m7i-flex.large (nginx + FastAPI + pipeline, no SSH port, IMDSv2) · S3 (models, data, results; run outputs expire after 7 days) · SSM Parameter Store (password) and Run Command (updates) · CloudWatch (logs, per-stage run metrics) · least-privilege IAM role.

Challenges we ran into

The "ground truth" was wrong. OpenCap clips labelled synced with mocap are offset per camera by several frames. Our results looked 14 frames late until we derived per-camera truth ourselves by projecting the motion capture into every camera. Against that truth our synchronization was within one frame; the dataset's own clips were within one frame only 31 % of the time.
A silent frame-rate bug. Triangulation assumed 30 fps when the input carried no frame rate, mis-scaling the smoother on 60 fps phone video. It now refuses to guess; that fix alone cut acceleration error from 2.49 to 1.44.
Walking fools synchronization. One step looks like the next, so a one-gait-cycle shift (about 60 frames) fits almost as well as the truth. We detect the ambiguity and report it instead of hiding it.
Small, portrait subjects. The person detector struggled on 720×1280 phone video of a distant person; a motion-based crop tracker fixed it.
No GPU in the cloud. Everything had to run on CPU, which is exactly where OpenCV 5's DNN engine beat ONNX Runtime.

Accomplishments that we're proud of

Real phones with lab ground truth (OpenCap LabValidation: 5 iPhones, 60 fps, public and consented). Scored against marker motion capture at zero time shift, with our own calibration and synchronization, next to OpenCap (Stanford's published smartphone system) on the same videos.

The error metric is the per-frame RMS joint error over 12 joints, after rigid alignment:

$$ e_t = \sqrt{\frac{1}{12}\sum_{j=1}^{12}\left\lVert R_t \hat{p}_{t,j} + T_t - p_{t,j}\right\rVert_2^2} $$

Subject 4, squats (held out)

Metric	AIMoCap (5 cams)	OpenCap (5 cams)	OpenCap (3 cams)
Joint error, median (mm)	41.4	40.4	39.2
Acceleration error (mm/frame²)	0.98	1.46	1.74
Jitter on still frames (mm/frame²)	0.44	1.24	1.42
Fast-motion speed ratio (1.0 = exact)	1.04	1.02	1.05

Across all recorded trials

Metric	AIMoCap	OpenCap
Median joint error (mm)	41–51	40–46
Acceleration error (mm/frame²)	0.98–1.72	1.07–1.99

Still-frame jitter is about 50–65 % lower than OpenCap on squats, synchronization is within 1–2 frames (1/60 s) wherever the reference is unambiguous, and knee flexion is within 3° of the lab system.

Studio datasets

CMU Panoptic and AIST++ have research-only licences, so we report numbers only, no footage. All results are on held-out clips with 3D ground truth.

Dataset	Median 3D error	Coverage
CMU Panoptic	20.6 mm	100 %
AIST++ (break dance)	23.7 mm	100 %

Change versus our previous pipeline, with no loss of median accuracy or motion speed:

Stage	Measure	Change
3D triangulation	Body jitter	−48 % to −57 %
3D triangulation	Fast-motion jitter	−32 % to −48 %
2D body	Jitter	−35 % to −40 %
2D fingers	Error / jitter	−11 % / −16 %
2D face	Error / jitter	−15 % / −37 %

Every change was accepted only on held-out clips, against gates fixed before testing.

Plus a full working cloud product: HTTPS demo, live runs, per-stage telemetry, one-command deploy and teardown.

What we learned

Measure the measurement first. Most of our "accuracy problems" were evaluation problems: a wrong reference, a wrong frame rate.
For animation, smoothness beats a millimetre. We are 1–5 mm behind OpenCap on median error and higher on worst-case error, but clearly smoother – and jitter is what viewers notice first.
OpenCV 5 is a complete deployment runtime. One library covered calibration, neural inference and geometry, and running it on CPU made a GPU-free cloud demo possible.

What's next for AIMoCap: Motion Capture from Any Camera

Guided recording in the app and consented uploads of your own videos.
A GPU option for full-length clips in minutes.
Better retargeting onto game rigs (foot orientation in turns), straight into Unreal and Unity.
A close-up camera for facial expressions.
Larger evaluation across more people and faster motions.

Data: OpenCap LabValidation (Uhlrich et al., PLOS Computational Biology, 2023, Apache-2.0; participants consented to public sharing). CMU Panoptic and AIST++ used for evaluation only.