punching-face

Inspiration

During this hackathon, Akash really wanted to punch his teammate, Mark, in the face. Unfortunately, Mark's skills were deemed too valuable to be wasted in a hospital. So, we came up with an alternative.

We built a low-latency image-3D pipeline with Astra 6.0, designed intelligent deformation for 3D human meshes, placed a multimodal Qwen model inside the head, added real physics, and BOOM — you can now punch whoever you want.

Another source of inspiration was this Instagram Reel: https://www.instagram.com/reels/DQHLEK_k5ov/

This project was built with love ❤️

What it does

Scan a person Our app allows you to pass a video or images, or just via your webcam. Once enough keyframes have been extracted, our model samples up to 220 frames and head-masks locally using MediaPipe's selfie segmenter. We then start a pipeline to build a high-quality 3D model via the segmented frames.

Building the model COLMAP maps video to get structure from motion, and reliable facial landmarks are pinpointed and triangulated, which is then mapped to a MakeHuman head fitted with a regularized global warp. Then, we add Catmull-Clark subdivision on top to ensure the facial loops, ears, skull and neck are smooth. Then we fit hair to the captured silhouettes. We bake the texture from photographs, matching the exposure and depth visibility across captures.

Generating the model The camera and captures only show the front of your face, so the back of the skull is inferred. To do this, we used gpt-6-astra to obtain masked crops and return parameters inferred from various facial dimensions, provided from the front. We then begin interpreting hair colour, flow, and hairline; eyeglasses; temple contour; and more, which our local mesh builder then executes. If the user decides so, we can also forward this model generation to a Meshy API and generate a mesh via Meshy instead of our pipeline.

Punching MediaPipe tracks your hands at 30fps. A hit drives a physics Newton1.6 model to simulate soft tissue on the CPU. With the various parts of the model rigged, we simulate different stiffnesses for cheeks/lips and for the nose and forehead. On top of the local tissue response to the punch, we add cheek compression, a lateral mouth pull, and other various facial features so the whole face silhouette follows the hit.

Punch. MediaPipe tracks your hands at ~30 Hz. A hit drives Newton 1.6 / Warp 1.17 running actual tetrahedral FEM soft tissue on the CPU: three particle layers (outer skin, soft tissue, fixed inner support) with different stiffnesses for cheeks/lips, nose and forehead, a kinematic fist collider, and a backtracking safeguard against inverted tetrahedra. On top of the local tissue response, a landmark-fitted impact rig adds cheek compression, lateral mouth pull, jaw shift and asymmetric eyelid squeeze, blended into the skull so the whole jaw silhouette follows the hit. Hold the peak deformation, slow it down, or turn off head recoil to isolate the surface.

Spar. The Cornerman is a Qwen3.5-Omni real-time session over a WebSocket: 1–2 fps downscaled frames from a worker, 20 ms PCM16 mic audio from an AudioWorklet, and a live feed of punch telemetry all go up; text, speech and tool calls come back. It sees your guard drop, hears you say "hold", and talks over the top of your combo — and you can talk over it, because barge-in cancels the response mid-sentence. Function calling lets it drive the app: change the view, soften the target, call a replay.

Arena. Go live and the 3D head is published to a LiveKit room. Anyone who scans the QR can punch it from their own phone — their hand tracking runs on their device, ~60 bytes of punch event crosses the network, and the hit lands on the same real physics with per-person scoring.

How we built it

Four processes, deliberately separated:

Browser — Three.js render, MediaPipe hand tracking, the React/Vite shell, frame worker and audio worklet.
server.py (:5174) — capture, reconstruction jobs, key custody, the Meshy backend.
physics_server.py (:5175) — Newton/Warp sessions, stepped at 30 Hz.
omni_relay.py (:5177) and sponsor_server.py (:5176) — the OMNI relay and LiveKit tokens.

Everything binds to loopback. No API key ever reaches the browser: keys live in .env or .local/secrets/*.json with owner-only permissions, and only a relay process ever holds one.

The split that mattered most is edge vs cloud. Perception, physics and a cached impact grunt are all on-device — contact to physical reaction is under 50 ms. The model's line arrives about a second later, on top of a reaction that already happened. OMNI runs beside the pipeline and never inside the render loop.

Sentry is wired as the flight recorder across all of it, because the defining failure mode of this project was silence: jobs died inside subprocesses and the UI showed nothing. sentry-trace and baggage are passed through the subprocess environment and continued in the child, so one click is one trace: browser → HTTP handler → pipeline subprocess → every individual pipeline stage. Plus structured logs carrying the trace id, gen_ai.* spans for every OMNI and OpenAI call (tokens, first-token latency, frames sent), Session Replay, and profiling on the Python services. Replay blocks all media, so the webcam is never recorded and the 3D canvas is never captured — asserted by a test, not by intention.

There is also a second reconstruction engine: Meshy's multi-image-to-3D API, which gets the four most informative cropped views and returns a GLB through the same import path. One scan can hold a model from each engine, so you can flip between the local pipeline and the cloud result without rebuilding anything.

Challenges we ran into

No CUDA, no Metal compiler. The whole thing was built on an M5 Pro with Command Line Tools only, so nvdiffrast, pytorch3d CUDA ops, gsplat and every build-time-Metal package were off the table from hour one. That constraint is why the physics is a real CPU FEM solve instead of a GPU library call — and, in the end, why it runs anywhere.

We had to abandon Gaussian splatting. The original plan was splat-based reconstruction. Our own quality gates refused the output: median splat thinness was 0.2–0.4, which means blobs rather than surfels, and meshing from Gaussian centres produced garbage. We pivoted to the photo-mesh + MakeHuman template pipeline within hours. Keeping gates that were willing to fail us was worth more than the work they invalidated.

Two Python interpreters that cannot share a process. open3d==0.18.0 pins us to Python 3.9–3.11; Newton/Warp needs 3.10+. And importing pycolmap and open3d in one process on macOS is a hard native crash from conflicting OpenMP runtimes. So the project runs separate virtual environments with separate requirements files, and a checker script that fails if either drifts.

Silent, confident wrongness. Self-calibrated intrinsics on our synthetic fixture recovered a focal length of 1701 px where ground truth was 1250 px (+36%), with radial k = -2.48 on a distortion-free render — while cheerfully reporting 0.41 px reprojection error across 40/40 views. Every gate passed. That one is written up in the repo as a trap for whoever touches it next, because the numbers lie.

Latency, in the only place it is allowed to matter. A punch that waits on a network round trip is not a punch. Everything that must be instant is local and cached; everything that can be late is late.

Accomplishments that we're proud of

The physics is real. Actual Newton 1.6 / Warp 1.17 with tetrahedral FEM, positive-volume enforcement and genuine recovery to an unchanged rest mesh. The physics label reports the engine that actually ran, and a failure stops new contacts rather than quietly falling back to a fake solver.
Contact to reaction in under 50 ms, with the AI layered on top rather than in the way.
We made the pipeline 2.5× faster without changing a single output — 309 s serial to 122 s accelerated, verified identical. The accelerator fingerprints the functions it patches and switches itself off if one is edited out from under it.
Honest scope, enforced in code. The build refuses to call front-only coverage a 360° scan. AI priors cannot touch measured geometry. Verified and unverified claims are tagged as such throughout the repo.
Multi-device sparring that survives a dev reload — the host rejoins under the same identity, with no ghost participant and no duplicate video track, and guests never notice.
A test suite that grew under pressure: 55 JS tests plus Python suites for the relay, tokens, Sentry wiring, the Newton solver and the photo pipeline, including hook guards that fail loudly if a cross-file integration point is deleted during a merge.

What we learned

Build the gate before you build the thing. Our quality gates are the reason we found out splatting would not work in hours instead of on Sunday morning. They cost us a plan and saved us the hackathon.

Reprojection error is not truth. A pipeline can be internally consistent, self-congratulatory and completely wrong. The only reliable check is ground truth you constructed yourself.

Latency budgets are architecture, not optimisation. Deciding up front which 50 ms were sacred determined the entire process layout — what runs on device, what streams, and what is allowed to be a second late.

Multimodality has to be forced by the situation to feel like anything. A coach you type at is a chatbot. During a round your hands are up and your eyes are on the target, so voice is the only possible input — and "was that hook better?" is meaningless to a model that did not watch the hook. The modalities are not features here; they are the only way the interaction exists at all.

Observability changes what you build, not just what you fix. Once one click was one trace, the expensive parts of our own system stopped being a matter of opinion.

What's next for punching-face

Close the calibration hole. Supply known intrinsics instead of estimating them, and add a regression test that checks recovered focal length against the fixture's stored ground truth.
Move capture review into the loop. The capture director already knows what is wrong with the frames; it should say so while you are recording, out loud, instead of after the build.
Real physical validation. The tissue layers, materials and collider are estimated — expressive, not measured. Tuning them against actual soft-tissue data would make this something other than a very convincing toy.
Auto-rigging on the demo path, so a fresh scan is punchable without a pre-rigged mesh.
Get the Newton solve onto the GPU on hardware that has one, and push the tetrahedral resolution up.
The non-joke version. A stress-release bag that watches your form, paces your breathing and knows when to tell you to stop. The safety prompting is already there, and the coach already refuses to encourage hitting a person.