PuppetMaster

What it is

PuppetMaster is a browser-based motion-capture “studio” that tracks your pose with the webcam, drives a stylized marionette character on a virtual stage, and lets you record, playback, smooth, and publish animations—without leaving the tab.

What inspired us

Classic creative tools (especially the era of timeline-first, stage-based animation) made animating feel tactile and immediate. We wanted that same “hit record and perform” energy, but with your body as the rig—no keyframes required to get something expressive on screen.

How we built it

Pose estimation: MediaPipe Pose Landmarker (lite model) runs in VIDEO mode with GPU delegation, pulling frames from getUserMedia.
Rendering: Two HTML5 Canvas surfaces—MoCap Studio (camera + skeleton overlay) and Stage (procedural puppet: limbs, torso, head, “hinge” joints).
Timeline: Each recorded frame stores landmarks + high-resolution timestamps; playback advances the playhead by elapsed time so motion stays tempo-faithful.
Smoothing: Optional 7-frame moving average over landmark coordinates to reduce jitter before export or playback.
Publish: JSON export for data pipelines, plus a self-contained .html player that embeds the timeline and replays the same character drawing—easy to demo or share.

Smoothing is a simple temporal filter: for each joint, we average coordinates over a sliding window of frames (here, window size (n = 7)), e.g. for coordinate (x) at frame (i):

[ \bar{x}i = \frac{1}{n} \sum{j=i-h}^{i+h} x_j, \quad h = \lfloor n/2 \rfloor ]

What we learned

How to keep 30fps-style loops responsive: heavy work stays in refs and requestAnimationFrame, with React state updated only where the UI needs it (e.g. throttled FPS / frame count).
Canvas is enough for a polished hackathon demo when the “character” is vector-style geometry driven by landmarks.
Export-as-HTML is a powerful hackathon pattern: judges and friends can open a file offline and still see the full demo.

Challenges

Jitter vs. responsiveness: Raw pose landmarks shake; smoothing helps but trades a little latency and “snap” for stability.
Browser permissions & init: Camera access and WASM/model loading need clear loading and error states.
Mirroring & coordinates: Mapping normalized landmark space to canvas space (including horizontal flip for selfie view) has to stay consistent across live view, stage, and exported player.