FieldSight · Real-time vision on any Arm board, no GPU

Inspiration

Every Physical AI project we've deployed ran into the same wall: the hardware bill. Putting vision AI in a parking garage, on a factory line, or in a surveillance camera usually means buying a Jetson or a dedicated NPU board for every single unit. So the AI never ships, or it ships as a stripped-down version of what we actually wanted to run. Meanwhile the boards are already out there. There are Arm CPUs in nearly every camera, gateway, and controller in the field, sitting mostly idle while frames get shipped off to a server 100-500ms away. We kept asking: is the CPU actually the bottleneck, or is it just that nobody optimized for it? Working on underground EV fire detection made this concrete - that's a job where a half-second of network latency matters and the camera is already mounted. So we set out to make an low cost Raspberry Pi 5 run real-time object detection with no GPU and no NPU, and to prove it with numbers anyone can reproduce.

What it does

FieldSight makes a plain Arm board see in real time. Point it at a camera on a Raspberry Pi 5 (or any Cortex-A board) and it runs YOLO object detection on-device at 3×-6x the frame rate of a standard PyTorch/ONNX pipeline - no GPU, no NPU, no accelerator stick. Detections are drawn live and fire as events, so only results leave the device, not video. It ships as three things: A real-time demo app - camera in, detections out. We demo intrusion detection and early smoke/flame detection for indoor parking garages, both running on the same board. An optimized model pack - baseline vs. optimized YOLOv8 artifacts, 60–75% smaller on disk and in memory, with an mAP report so you can see exactly what the compression cost. A benchmark harness - one command measures FPS, per-frame latency, memory, CPU utilization, and power on your own board and writes out the comparison table and graphs. That last part is the point. Every number we claim, you can regenerate on your hardware in a single run.

Two measured things on one Raspberry Pi 5, clock pinned at 1500 MHz. Detection: YOLOX-Nano runs at 45.68 ms per frame at 416x416 on four Cortex-A76 threads with stock ONNX Runtime and no accelerator. YOLOX-Tiny is 131.61 ms on the same setup. Single thread, those are 115.61 ms and 407.51 ms. Control policy: a Decision Transformer costing 4777.07 us per step, distilled, stripped of its runtime, then rewritten with NEON intrinsics.

4777.07 us to 3.67 us on one core: 1302x. The file goes from 3.48 MB to 75 KB. The repository also ships the harness those numbers came from: it pins every core, waits for the SoC to cool, interleaves models across repeats, and records the clock at both ends of every timing window.

How I built it

The policy path is four independent pieces, each measured on its own so the 1302x can be attributed rather than just quoted: distillation to a small MLP, ONNX export, a plain C rewrite with weights inline and no framework, then a NEON version of the same arithmetic. Both C builds are one gcc command and link nothing but libc. The detection path uses public YOLOX ONNX weights and stock ONNX Runtime, so the numbers are what a developer gets on day one rather than what a tuned stack can be pushed to.

The harness came last and should have come first.

Challenges I ran into

The first pass did not lock the clock. Four models measured back to back on an idle Pi 5 under the ondemand governor ran at 2400, 2366, 1500 and 2146 MHz. Every ratio in that table was wrong. I only caught it because one model got measured twice and disagreed with itself. Locking the frequency, gating on temperature and interleaving the models brought repeat-to-repeat spread down to 0.05%, and the harness now stores the observed clock next to every number so a drifted run shows up in the artifact instead of in the conclusion.

The second was assuming threads help. They do for the detector: 2.53x for Nano, 3.10x for Tiny. For the 75 KB policy the pool costs exactly what it saves, 25.04 us on one thread against 25.07 us on four. Below roughly a hundred microseconds of work per call there is nothing left to split.

Accomplishments that I'm proud of

Getting a policy step under 4 us on a single Cortex-A76 core, and being able to say where each factor of the 1302x came from instead of reporting one number.

And the measurement setup catching my own mistake before it reached a README.

What I learned

Size and latency are not one axis. Distillation moved both. Vectorization moved latency alone. Threading moved latency only above a size threshold and did nothing at all below it.

I speedup without the clock it was measured at is not a measurement.

The step everyone skips is worth more than expected: dropping the runtime and hand-vectorizing the same arithmetic was 6.8x on top of the distillation, on a model small enough that the framework around it cost more than the math inside it.

What's next for fieldsight-arm

More boards, and a public leaderboard. The harness is already board-agnostic - the obvious next step is collecting results across Rockchip, Qualcomm, Ampere, and the rest of the Cortex-A range into an open comparison of what each board can actually sustain for real-time vision. Right now a developer choosing hardware for an edge AI product has no good reference for this. We'd like FieldSight to be that reference. Turn throttling into a scheduling input. We currently report thermal behavior. The better answer is to react to it: adapt resolution, frame rate, or model variant as the board heats, so a camera degrades gracefully instead of falling off a cliff at minute three. Thermal-aware scheduling is the piece that separates a benchmark from a device you can leave in a garage for two years. Use the accelerators when they're there. We stayed CPU-only to prove the point. The engine can already split a graph across CPU, GPU, and NPU, so the next release lets boards that have silicon use it - same API, better ceiling. Close the loop with Arm64 in the cloud. Running the identical model on Graviton and profiling it with Arm Performix would let us publish a single cost-and-latency picture spanning edge to cloud on one architecture. That's the comparison teams actually need when deciding what to run where. Beyond detection. Segmentation and pose next, then the harder target: small vision-language models on-device, so a camera can answer questions about a scene instead of just labeling boxes. That's where "intelligence in the field" stops being a slogan. And into the field. The garage fire-detection and license-plate pilots are the real test. Benchmarks are how we got here; uptime in a concrete basement with bad lighting is how we'll know it works. Per-layer profiling of the NEON kernels to find w here the remaining 3.67 us actually goes, and the same protocol run on Cortex-A53 and A78 to see which of these conclusions belong to the A76 rather than to Arm.

Built With

• aarch64
• arm
• c
• neon
• onnx-runtime
• python
• raspberry-pi
• yolox