Motion Patch

Automated mocap cleanup that writes Blender scripts instead of guessing coordinates.

Comment

Inspiration

On a virtual production stage, the director's priority is the actor's performance. When a stunt double lands a fall or an actor delivers a line in a tracking suit, that physical timing is what the production buys.

When the shoot wraps, problems show up. Markers drop out behind props, cameras lose tracking, and boots slip on smooth stage floors. Visual effects teams spend days hand-cleaning f-curves in animation software before that take can drive a digital double.

Most AI tools invent new motion from scratch, which wipes out the take the director approved. We built MotionPatch to act as an assistant technical director instead. It finds tracking glitches, writes Blender Python scripts to repair the curves, and leaves the actor's performance intact.

What it does

MotionPatch cleans motion-capture takes while keeping animators in control:

  1. Scans raw takes. A kinematic engine analyzes raw BVH takes across seven physical domains, flagging foot sliding, marker jitter, root drift, and Euler gimbal flips.
  2. Retrieves past fixes. A ClickHouse vector database supplies historical repair scripts and filter frequencies used on similar stunts.
  3. Dispatches specialist agents. Google Cloud ADK and Gemini 3.7 Flash spawn targeted agents assigned to specific joints and frame intervals.
  4. Generates verified Blender code. Agents write Blender Python (bpy) scripts. An Abstract Syntax Tree (AST) validator checks that code targets assigned bones and blocks unsafe imports.
  5. Executes in Docker. Headless Blender runs the script in an isolated container and bakes the corrected animation curves.
  6. Provides white-box review. Animators inspect original and repaired skeletons side by side in a 3D web viewport, edit the Python script in a Monaco drawer, and export the approved BVH file.

How we built it

1. Kinematic detection engine

Before calling an LLM, the backend runs a forward kinematics pass in pure Python (backend/detector.py).

The engine unwraps Euler angles frame to frame to stop false spikes near 180 degrees:

$$ \theta_{\text{unwrapped}}[t] = \theta_{\text{unwrapped}}[t-1] + \text{mod}(\theta[t] - \theta[t-1] + \pi, 2\pi) - \pi $$

It detects joint jitter using rotational acceleration Median Absolute Deviation across the take:

$$ \text{MAD} = \text{median}\left(\left|a_t - \text{median}(A)\right|\right) $$

It flags foot sliding during ground contact frames ( y \le y_{\text{contact}} + \epsilon ) when horizontal slip velocity exceeds physical thresholds:

$$ v_{\text{slip}} = \sqrt{\dot{x}^2 + \dot{z}^2} $$

2. ClickHouse vector memory bank

Defect profiles are converted into 768-dimensional normalized vectors using Google Cloud Vertex AI text-embedding-004. A FastMCP server queries ClickHouse using cosine vector similarity to retrieve proven repair scripts:

$$ \text{similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}|_2 |\mathbf{v}|_2} $$

3. Agent orchestration and sandboxing

Gemini 3.7 Flash runs on Google Cloud Vertex AI through the Google Cloud Agent Development Kit (ADK). The agents generate Blender Python (bpy) scripts that apply Butterworth filters, Euler unwrap passes, or floor-contact constraints.

An AST validator (validate_qa_script) inspects the syntax tree before execution to ensure the script references assigned bones and contains no unauthorized modules like subprocess or socket. Validated scripts run inside a headless Blender Docker container.

4. Web studio interface

Built with React 19, TypeScript, and Three.js via @react-three/fiber, the frontend gives artists:

  • A split 3D viewport for side-by-side or ghosted comparison.
  • A timeline scrubber marking defect locations.
  • Voice input via the Web Audio API for on-set director notes.
  • A Monaco editor drawer for modifying Python scripts directly.

Challenges we ran into

  • Euler angle gimbal flips. Near ( 90^\circ ) of pitch, Euler channels snap 180 degrees in a single frame. Simple channel checkers flag these as false velocity spikes. We built forward kinematics matrix chains that compute true 3D spatial points ( [x, y, z]^T ) for every bone before evaluating errors.
  • Intentional acting versus tracker noise. Hard stunt landings look like noise to generic smoothing filters, which destroys the scene's impact. We computed whole-take baseline metrics to target only genuine statistical outliers.
  • Sandboxed execution. Running agent-written code in a studio pipeline risks crashes or security leaks. We enforced a two-tier guardrail: an AST import validator and an isolated Docker container with execution timeouts.

Accomplishments that we're proud of

  • 100% test pass rate. 85 backend tests in pytest and 70 frontend vitest tests pass cleanly.
  • Zero comments policy. Backend code strictly adheres to a zero-comments rule verified by automated AST scanning.
  • Zero false positives on benchmarks. The kinematic detector produced zero false positives across clean reference clips from the Carnegie Mellon University mocap database.
  • True white-box control. Animators can read the exact Blender Python code, adjust numeric parameters in the browser, and inspect results in real time.
  • Full pipeline integration. Connected Google Cloud ADK, Gemini 3.7 Flash, ClickHouse FastMCP, headless Blender in Docker, and a real-time Three.js viewport into a cohesive workflow.

What we learned

  • Code generation beats coordinate prediction. LLMs struggle to predict floating-point 3D coordinates without drift, but they generate clean procedural Python scripts that fit existing VFX pipelines.
  • Kinematic pre-filtering saves money. Running local kinematics before calling Gemini cut prompt tokens by over 90% and gave the model exact frame ranges and joint names.
  • Inspectability earns trust. Animators reject black boxes. Giving artists an editable code drawer alongside 3D playback turned skeptical animators into active users.

What's next for Motion Patch

  • Multi-character and prop tracking. Expanding detection to two-actor stunt choreography and prop interactions.
  • OpenUSD export. Adding native USD export for direct handoff into Maya, Houdini, and Unreal Engine 5.
  • On-set camera monitor sidecar. Packaging the detector to flag tracking glitches on camera operator monitors during live takes.
  • Production database syncing. Connecting the batch engine to Autodesk Flow Production Tracking to log take quality automatically.

Built With

Share this project:

Updates

Submission history