Tactical Canvas

Inspiration

The FIFA World Cup is where the whole planet suddenly cares about a back line holding its shape. But watch how coaches actually teach it: either a 2D app they click through alone at a desk, or a magnetic whiteboard with arrows and zero data behind them. Neither is physical, and neither tells you whether the fix was actually right.

We wanted the thing you see in the analyst's dream: a coach pausing a real match, walking up to a wall-sized pitch, physically dragging a defender to where they should have been, drawing the covering run with their finger — and having an AI second opinion light up the space that just opened or closed. Tactical Canvas is that board.

What it does

A projector throws a top-down pitch onto a wall. A match replay plays on the coach's laptop, and the projected board shows the real player positions for that exact moment, driven by synchronized tracking data.

The coach pauses at a defensive breakdown, walks to the wall, and:

Drags a player's circle with their hand to a better position — grab is a pinch, release is a spread.
Draws passing and running lines with their index finger.
Toggles AI overlays — Voronoi pitch control, defender shadows, pass lanes, and "ghost" suggested positions — to compare what happened against what should have happened.

A dashboard shows the stats for that frame alongside the overlays, so a coach can argue their fix against the model's fix, on the wall, in front of the players.

How we built it

Three processes, on purpose — not five.

Frontend — a projector view and a dashboard view sharing one PixiJS (WebGL) pitch renderer and one WebSocket client.
FastAPI core — the single authoritative state machine. Clients send commands (SET_PLAYBACK_TIME, DRAG_PLAYER_MOVE, ADD_PASS_LINE, TOGGLE_OVERLAY…); the server owns state and broadcasts versioned snapshots. Every message carries a protocol version, scenario id, and sequence number so a stale drag or a reconnect can never corrupt the board. Analytics (pitch control, pass probability, compactness, suggested positions) are ordinary Python functions over current state — NumPy/SciPy in-process, no separate model service.
Vision worker — a separate process running OpenCV + MediaPipe Hand Landmarker. It does homography calibration, a pinch state machine with hysteresis, coordinate smoothing and jump rejection, and emits semantic events (touch_move, pinch_start…) — never raw landmarks — over a multiprocessing queue. It lives in its own process so camera capture and hand inference can never stall the WebSocket hub.

Sync is the quiet hard part: we drive playback off requestVideoFrameCallback so we sync on the frame that's actually on screen, map that media timestamp to a tracking frame server-side, and let the projector interpolate between broadcasts for smooth motion. Real Metrica tracking data (25 Hz, with true velocities) — not interpolated event data — because velocities are load-bearing for defender shadows and pitch control.

Challenges we ran into

Monocular touch. Distinguishing "pointing near the wall" from "touching it" from "dragging" needs depth, and we had one webcam. We solved the verb, not the depth: a pinch (thumb-tip↔index-tip distance, normalized by palm length) is unambiguous from any angle. The fingertip gives the cursor position; the pinch gives the intent. That single decision is what made a monocular rig viable.
The camera lied to us. Forcing 720p + MJPG gave us 4.8 fps — the format request was silently ignored, then the resolution blew past USB bandwidth. Letting the camera negotiate its own defaults gave 640×480 at ~30 fps. We also learned the backend hint matters: the wrong one took ~83 seconds just to open the camera.
Calibration under occlusion. The coach's own body blocks the projector while they reach for the wall. Getting a stable homography and keeping drag ownership per hand took hysteresis, smoothing, and a hard kill-switch for touch.
Cross-platform camera access. We developed on macOS and demoed on a Windows rig; AVFoundation vs DirectShow backends and macOS camera permissions each cost us real debugging time.

Accomplishments that we're proud of

A physical tactical board where a coach drags real players and draws with a finger — and it stays in sync with the video frame that's on screen.
One authoritative state machine driving both a projector and a dashboard with a versioned protocol, so the two views never disagree.
Making monocular hand interaction actually usable via the pinch insight, instead of waiting for a depth camera we didn't have.
Real tracking data and real analytics (Voronoi pitch control first) — the AI overlay shows a genuinely different fix than the coach's, which is the moment that sells the whole idea.

What we learned

Fewer runtime boundaries beats a clever architecture. Every process split costs sync bugs and startup pain; we paid for exactly one (the vision worker) and it was worth it.
Sync off what's on screen, not what you intended. Timers and framework state report intent; requestVideoFrameCallback reports reality.
Measure the camera, don't trust the API. Half our "performance work" was discovering that setting properties made things worse.
A guaranteed input fallback is worth more than another AI overlay. Mouse control that always works kept the demo alive while we hardened finger touch.

What's next for Tactical Canvas

Depth camera (RealSense / Kinect) to turn touch into true wall-plane contact — an upgrade our architecture already anticipates, not a rewrite.
More overlays: defender shadows and a pass-probability heatmap next, then gradient-ascent suggested positions on a movement budget.
Scenario library — save annotated moments to SQLite and replay a full teaching session deterministically.
The React + Vite port of the frontend (the renderer and WS client are already isolated for it) and TypeScript types generated from the Pydantic schemas.
Multi-hand, multi-coach interaction, and eventually live tracking so it works on training-ground footage, not just replays.