Masiez

Inspiration

Every drone-autonomy demo we watched needed three things nobody talked about: a server to run the model, a contractor to draw the boxes, and an operator who never lets go of the stick. Each one is infrastructure, not research.

So we asked how much of the loop actually fits in a browser tab. Not a toy renderer — the whole thing: the world, the pilot, the labels, the detector, and the drone flying itself toward a target it can see.

Tanks on an open field made a good test scene. One object class, brutal viewpoint changes, and a target that is either found or not.

What it does

Masiez is a real-time 3D drone battlefield that renders, flies and labels its own camera in a single tab.

Fly it. A quadcopter over a 4.4 km procedural terrain field, keyboard or gamepad, with a HUD that reports current draw, mode and heading.
Let it see. Toggle target mode and the camera's detections appear as boxes. Click one and screen-track autonomy steers the drone at that box. Pilot input always wins — any stick movement cancels the autopilot instantly.
Make it label itself. One keystroke writes the rendered frame and its boxes together. A generator produces 360 frames per run (300 train, 60 valid, one tank category) as an archive the training notebook reads without conversion.
Put the weights back. Trained RF-DETR runs in the tab as ONNX in a module worker. The menu switches between the software projection, a 41 MB femto model and a 36 MB atto model, so you can watch the learned detector disagree with ground truth on the same live frame.

Measured on this build: physics stepped at 250 Hz, a frame settles at 38 draw calls, terrain is one indexed mesh of 48,841 vertices and 290,400 indices, and the HUD read 144 FPS at 1280×720 on the dev machine. Boxes drawn by hand: zero.

How we built it

The renderer is hand-written WebGPU — no game engine, no scene-graph dependency. The flight model is C++ compiled to WebAssembly and advanced through an accumulator at a fixed 1/250 s, so a dropped frame resolves every substep instead of skipping physics.

Inference runs on onnxruntime-web with the WebGPU execution provider and a WASM fallback, inside a dedicated module worker so the render loop never waits on a model.

Training stays off-box on Kaggle: start from public COCO weights, fine-tune 40 epochs at batch 8 and lr 1e-4 on a single class, then export ONNX. The folder the game writes is the folder the notebook expects.

There is no backend. The product is the client; the earlier realtime-server work is parked on a backup branch as an alternative, not a dependency.

We also produced the artifacts to prove it works: a 28-second demo video, a poster, and a six-slide deck — all built from real captures, no generated imagery.

Challenges we ran into

Silent numerical failure. The fp16 export of the detector returned wrong boxes with no error at all — GridSample in fp16 on WebGPU. That is the worst class of bug: the model looks alive and is lying. The exported copy now keeps that op in fp32.

Recording the tab honestly. Canvas-only capture loses the HUD because the HUD is DOM. Playwright's built-in video wrote a codec the local ffmpeg could not decode. We drove the page over the DevTools protocol and pumped frames on a fixed interval — after two takes that ran about 1.5× fast because screencast bursts defeated a wall-clock gate.

Boxes that were not there. Early takes showed a clean sky and no detections. The overlay only projects when target mode is on; a gate in the code, not a bug — but it cost us two recordings to find.

Deck toolchain surprises. Alpha channels on shape fills were dropped, so scrim bands over photos rendered opaque; we dimmed the pictures themselves instead.

Losing the backend on purpose. Halfway through, the server was removed from the mainline. It forced the real decision: run the models in the browser or admit the demo needs infrastructure.

Accomplishments that we're proud of

The labelling loop closes with zero manual annotation, and the same projection that trains the model also scores it live.
Everything needed to run ships as static files — physics binary and both detector weights are in the repo. First flight is npm install and npm run dev, with no compile step.
An autonomy contract that never fights the human, which matters more than the mAP number.
Every figure in this write-up came from the running app or the source, and every deliverable was verified from the artifact itself: frames extracted from the finished video, the saved deck parsed back out.

What we learned

A browser is now a legitimate autonomy runtime. The interesting constraint is no longer "can it run here" but "which stage of the loop still needs a server, and why".
Ground truth by construction beats any argument about annotation quality. A simulator cannot mislabel what it placed.
Silent wrongness is worse than crashing. Validation has to compare outputs, not exit codes.
Verify the artifact you ship. The intermediate can look perfect while the file you hand over is broken.

What's next for Masiez

Close the dataset loop. Make the in-game export the notebook's default source instead of a manually uploaded copy.
Range-aware policy. The target-following policy needs range in its observation so loiter and engage behavior stops being hand-tuned; that means a matching training environment and loader.
CV Arena. A classic OpenCV pipeline competing against the learned detector on the same captures, so the comparison is method to method rather than model to nothing.
Tests and CI. Coverage over physics stepping, box projection and dataset export, run on every push.
HUD depth. A latency graph and a debug observation overlay so you can see what the pilot actually feeds the policy.
Optional multiplayer. Revisit the realtime backend and LiveKit as an opt-in teleop or race mode, from the branch that still has it.