The Matrix

The Matrix

Inspiration

AI agents reason well and act on nothing. We wanted the opposite constraint: three ~$50 robots, one camera each, no encoders, no IMU on the drive board, no lidar, one laptop, and a rule that the robot must never be left driving on a stale decision. The Matrix is what falls out of taking that constraint seriously: a hard real-time safety layer on the robot, everything else off-board, and a shared world model that three cameras write into.

What it does

Today the system is three Freenove 4WD ESP32 cars driven concurrently over Wi-Fi from one laptop, streaming acquisition-timestamped video, with per-frame AprilTag pose estimation and a fleet console that renders one shared map, a task tree and a Gaussian-splat reconstruction of the space.

Concretely, on hardware:

Three robots armed and driven at once from one keyboard (WASD / TFGH / IJKL), each on its own camera thread and its own 30 Hz command thread. One robot faulting never stalls another.
Every command is a 32-byte UDP packet bound to a boot nonce and a session token. The robot stops itself 300 ms after the last valid drive packet, on a FreeRTOS task that does not depend on the network or camera loop.
Every video frame carries the sensor capture time and the firmware boot ID, so the laptop can reject frames from a previous boot and time-align frames across robots to within a bounded clock error.
AprilTag detections per robot are fused into a camera pose, and robots that see each other's tags can be chained into one frame.

Autonomy runs where it can be measured: an EKF localizer scored against ground truth, a MuJoCo three-car simulation with a lens and sensor model, and an in-browser world model that explores with frontier allocation and A* over a 10 cm occupancy grid. Closing that loop on the physical cars is the next step, not a claim we make yet.

How we built it

Robot. Freenove FNK0053 kit: ESP32-WROVER with PSRAM, OV2640, PCA9685 motor driver on I2C at 50 Hz, 12-bit PWM. Pan/tilt servos are frozen at 90° so the camera mount is rigid. Firmware runs the motor and safety task at a 5 ms tick on core 1 at priority 4, independent of the Wi-Fi and camera handlers on core 0. The build is 80 % of flash and 18 % of static RAM.

Wire protocol (htnrobomaze/firmware/maze_robot/protocol.h, coordinator/networking/command_protocol.py). UDP 4210, exactly 32 bytes, big-endian, magic MZ01: robot id, kind, boot nonce, session token, sequence, robot-uptime stamp, left and right wheel as float32 in [−1, 1]. Kinds are HELLO, ARM, DRIVE, STOP, ESTOP. Arming needs zero wheels, a matching nonce and token, healthy motors and camera, and a stamp within −30 ms to +150 ms of robot time. DRIVE additionally needs a strictly newer sequence. The token increments on every disarm or e-stop, so packets in flight from a dead controller are invalid. STOP and ESTOP are accepted with stale session fields on purpose: anyone on the LAN can stop a robot, nobody can commandeer one. Telemetry is JSON every 100 ms with armed, e-stop, reason, RSSI, camera and motor health.

Camera. MJPEG over TCP 8080 at 320×240, JPEG quality 14, capped at 12 fps, double-buffered from PSRAM, always grabbing the latest frame. Each part carries X-Capture-Ms from the frame buffer's acquisition timestamp and X-Boot-Id. The laptop's freshness gate uses the capture time, not the receipt time, and drops frames whose boot ID disagrees with telemetry.

Clock sync (coordinator/networking/clock_sync.py). A read-only request/reply on UDP 4211 at 2 Hz. The mapper keeps eight samples, rejects round trips over 250 ms, adds a 1000 ppm drift allowance, and refuses to call a frame synchronized if the uncertainty exceeds 25 ms. Unsynchronized frames fall back to receipt time and are never used for cross-robot pose chaining.

Laptop control loop (coordinator/). Pygame at 60 Hz for real key-up and focus-loss events; a 30 Hz sender per robot; heartbeat and camera timeouts of 0.35 s; an input lease of 0.15 s. The config type rejects any control rate outside 20 to 50 Hz or any freshness limit looser than these, so the safety numbers cannot be misconfigured. There is no arm button: a neutral-to-motion gesture performs the ARM handshake, and a fault consumes the gesture, so holding a key through a recovery cannot restart motion. A per-robot OS file lock stops two dashboards from driving the same robot, added after three concurrent dashboards invalidated an acceptance run.

Localization (coordinator/localization/). tag36h11 via pupil-apriltags, one detector thread per camera on the newest frame. Pose per tag from solvePnPGeneric with IPPE_SQUARE, distortion-aware, keeping both solutions and flagging ambiguity when the second is within 10 % reprojection and more than 10° away. Gates: Hamming 0, decision margin ≥ 20, mean tag edge ≥ 20 px, reprojection ≤ 2 px. Multi-tag fusion is a maximal-clique consensus (Bron–Kerbosch) over the camera poses each world tag implies, gated at 0.15 m and 15°, weighted by margin and reprojection. Two equally supported contradictory cliques return no pose rather than a guess. Poses chain robot-to-robot over up to two hops with a 0.1 s maximum skew. A guided calibration tool captures at least 15 views of a 9×6 checkerboard and refuses to save above 1 px RMS.

Offline localizer (pipeline/localize/). Records 30 fps runs into run.json (schema thematrix.run/1) with motor duty and every tag detection, then runs an EKF on (x, y, θ): predict from a deadband duty model plus gyro yaw rate, correct with mapped tags as range-bearing landmarks under a χ² gate of 9.21. The /localize page replays a run with tag quads drawn on the video and the 2σ covariance ellipse on the map.

Reconstruction (pipeline/splat/). Sharpest-frame extraction, COLMAP 4.2 on CPU with a single OpenCV camera and sequential matching, then Brush (Metal) for training. A --lowres path switches to exhaustive guided matching, lowered SIFT peak threshold and domain-size pooling for the robots' 320×240 footage. The console renders the result with three.js 0.186 and Spark, with a fog texture over unexplored cells.

Simulation (pipeline/sim/). MuJoCo at 600 Hz physics with velocity actuators and elliptic friction cones, three cars with the kit's geometry (0.13 m track, 65 mm wheels, camera on a 0.15 m mast), a 6 × 4 m room with movable tagged box, and a camera model that adds barrel distortion, 20 ms motion blur from the gyro, 30 ms rolling-shutter readout, locked AE/AWB, shot and read noise, JPEG at quality 82 and 1.5 % held frames. A small server streams MJPEG feeds and pose events to the /sim page.

Fleet console (dashboard/, Next 16, React 19, zustand, React Three Fiber). The world model is browser-free TypeScript: a 60 × 40 grid of 10 cm cells, a 28-ray sensor model at 10 Hz from each pose, frontiers recomputed at 2 Hz, decisions at 4 Hz, A* over (cell, heading) with turn penalty and clearance inflation, and a single-leg auction whose utility is unknown cells seen minus distance with a zone bonus and a claim discount near another robot's target. A console language parses explore, move box to x y, goal, speed, pause, seek and so on into the same task tree the demo uses.

Challenges we ran into

No odometry, and a wrong pose is worse than none. The kit has no encoders. We put pose on wall tags and made every stage refuse rather than guess: ambiguous IPPE solutions are flagged, contradictory tag cliques return nothing, unsynchronized frames cannot chain, and a world-tag or camera-mount entry with measured: false yields no transform at all instead of an identity. The null placeholders in config/world_tags.yaml are still null. Tag IDs decode on hardware; metric world localization on hardware is not yet claimed.

Wireless control that fails closed. A dropped link must not leave a robot on its last command. The 300 ms watchdog lives in firmware on its own task, HELLO packets deliberately do not refresh it, the session token invalidates in-flight packets on every disarm, and the close path sends ESTOP three times. What we have not measured is physical stopping time; the 5 ms and 300 ms figures are scheduling targets, and a hard guarantee would need a hardware cutoff.

Timestamps are the whole problem. On a phone hotspot, stamping a frame at HTTP receipt is the dominant pose error for a moving robot. Capture-time headers, boot-ID rejection and the bounded clock mapper exist because of this.

Robot footage barely reconstructs. COLMAP on 320×240 MJPEG registered 2 of 180 frames with default settings. The low-resolution SfM path registers 180 of 180. That one comparison decided the pipeline's defaults.

Detection is not navigation. We ran YOLO11n and Depth Anything V2 Small live on a robot feed, and the overlay is explicitly display-only: it never touches motor commands, and it lives on an unmerged branch. A bounding box does not tell you whether the floor ahead is drivable, and the depth score is relative, not metric.

Accomplishments that we're proud of

98 passing Python tests plus native C++ safety and protocol tests compiled under -Wall -Wextra -Werror; three robots flashed with verified hashes and real telemetry.
Three-robot concurrent teleop with independent failure domains and a lease that makes two controllers on one robot impossible.
A localizer scored against ground truth: on a synthetic 450-frame run with 10 % motor calibration error and gyro bias injected, 609 tags fused, 0 rejected, RMSE 1.57 cm and 0.19°, worst 4.25 cm.
Real reconstructions: 179 of 179 frames and 1.08 M gaussians from a 4K phone clip, 522 k from the same walk downsampled to 640×480 MJPEG, and 118 k from the robot's own 320×240 camera.
A fleet console whose map, fog, frontiers and reconstruction are always computed from where cameras actually pointed, even when the demo's poses are scripted to a 27.3 s overhead video.

What we learned

The robot cannot wait for the laptop, so the layers have to run at different rates and fail independently: a 5 ms motor task, a 30 Hz command loop, 12 fps video, 2 Hz clock probes, and a planner that is allowed to be slow. Anything uncertain, a depth model, a reconstruction, a language model, belongs outside the control loop where the worst case is a missing caption, not a wall.

The other lesson is to refuse instead of guess. Every silent fallback we removed, a default focal length, an identity mount transform, a receipt-time frame stamp, would have produced a confident, wrong robot.

What's next for The Matrix

The design is written and partly compiled; the hardware loop is not closed. In order:

Measure the arena: tag poses, camera mounts and intrinsics per robot, so measured: true and metric localization on hardware becomes a claim.
Drive the EKF and the frontier allocator from live tag poses instead of recorded runs and scripted demos.
The four-board firmware in firmware/ (200 Hz yaw-rate PID control board over ESP-NOW, VGA vision board over UDP, MPU-6050 IMU board, USB gateway as the common timebase) compiles against ESP32 core 3.3.12 but has no hardware bring-up yet.
Motion-stereo depth from the slow, continuous 0.10 to 0.15 m/s cruise, as specified in the perception doc, to replace tags for obstacle geometry.
Measure physical stopping time and the end-to-end capture-to-actuation latency the test plan specifies.

Long term the platform is for spaces where sending a person first is slow or unsafe: search and rescue, inspection, warehouses. The honest version of the claim is established methods, composed on cheap hardware with one camera as the only exteroceptive sensor, running continuously and measured against ground truth.