Edge Triage: A 0.5B LLM Watching Sensors on a Raspberry Pi 5

Inspiration

Edge AI is usually pitched as "buy an NPU." We wanted to know what a plain ARM CPU can actually do — no accelerator, no GPU, no cloud fallback. A Raspberry Pi 5 has four Cortex-A76 cores and a 4W power envelope. Can that run a real language model inside a hard real-time budget?

The use case we picked is unglamorous and real: an IoT sensor stream — temperature, humidity, people count — where something occasionally goes wrong and somebody needs to be told what and how badly. Threshold rules are fast but stupid; they fire on every open window and every space heater. An LLM understands context but is far too slow to run on every reading. That tension is the whole project.

What it does

A hybrid two-stage pipeline. Stage one is six threshold rules that run in under 0.1 ms and clear 82% of all sensor windows — no model involved. Stage two escalates only the ambiguous remainder to Qwen2-0.5B, quantized to Q4_K_M (380 MB), running locally through llama.cpp. The model emits a structured JSON verdict: alert (bool), reason (string), severity (low/medium/high).

There's a second gate inside stage one: windows that trigger only low-severity rules (people count alone) alert directly without ever waking the model, skipping roughly another 30% of would-be LLM calls.

Measured on a Raspberry Pi 5 over 4 hours of sensor data — 480 windows, 55 threshold triggers, 41 LLM escalations:

p50 LLM triage 1,780 ms p95 LLM triage 1,826 ms p99 LLM triage 1,841 ms Budget misses 0 of 41 (2,000 ms budget)

How we built it

The interesting part isn't the architecture, it's the optimization chain. We started at 5.8 seconds p99 and got to 1.84 seconds. Every step was measured independently:

FP16 -> Q4_K_M quantization 949 MB -> 380 MB 2.5x memory n_threads=4 -> n_threads=1 4.0 -> 15.1 tok/s 3.8x generation max_tokens 128 -> 25 ~8.5s -> ~1.6s per call 5.3x per call lazy load -> eager load + warmup 5,833 -> 1,841 ms p99 3.2x cold start generic pip -> native NEON build 4.0 -> 28.4 tok/s 7x prompt processing hybrid threshold gate -> ~324 ms effective avg 5.5x system level

Two of those are counterintuitive enough to be worth calling out.

The thread count is backwards from what you'd expect. In llama-cpp-python, n_threads=1 is 3.8x FASTER than n_threads=4 on this hardware. The Python callback fires per generated token and re-acquires the GIL each time; with four threads that contention on ARM's smaller cores costs far more than the parallelism wins. Native llama.cpp doesn't have this problem — llama-bench hits 28.4 tok/s at 4 threads. It's purely a binding-layer artifact, and you'd never find it without profiling both paths.

The prompting strategy also inverted. Instruction-style prompts fail at 4-bit — the model drifts, wraps JSON in prose, truncates mid-object. Switching to few-shot completion, where the prompt ends mid-structure and the model simply continues the pattern, produced 100% valid JSON across every run. No LoRA, no fine-tuning, no RLHF. We rewired the model's behavior entirely through prompt shape, which matters a lot when your whole deployment budget is 380 MB on a device with no training capability.

Challenges we ran into

The cold start nearly sank the latency budget. Our first honest benchmark showed p99 at 5,833 ms against a 2,000 ms target — a catastrophic miss. It turned out to be a single outlier: the very first LLM call, which pays for loading the model from disk. Everything after it ran at 1.3-1.8s. The fix was eager loading plus a warmup inference at startup, and the warmup has to use the full few-shot prompt, not a trivial one, because llama.cpp defers some internal allocations until it sees a realistic workload. A short warmup string leaves the allocation cost sitting in your first real request.

We also chased the on-board Hailo-8 accelerator for a while before establishing that it cannot help here at all. It's a CNN inference engine — no attention primitives, no KV cache, no autoregressive decode path. Not "slow for LLMs," structurally incapable of them. We documented this as a negative result because the marketing around edge AI accelerators strongly implies otherwise, and the next person deserves to skip that week.

Accomplishments that we're proud of

Zero budget misses across 41 consecutive LLM escalations on a $80 general-purpose computer, with a 500M-parameter transformer in the loop. Not a demo that works once — a p99 number over a sustained run.

And the honest negative result on the NPU. It would have been easy to quietly drop it.

What we learned

Optimization on ARM is mostly about finding the thing that isn't the model. Quantization got us 2.5x on memory, which everyone expects. But the GIL discovery, the token budget, and the cold start together account for far more of the actual latency win — and none of them are model problems. The largest single gain, 5.5x at the system level, came from the architectural decision to not call the model at all 82% of the time.

The cheapest inference is the inference you skip.

What's next

Real sensor hardware instead of synthetic streams — the generator models four anomaly types faithfully, but it isn't a warehouse. Beyond that: a quantization sweep now that Q4_K_M is proven as a baseline, batching for multi-room deployments, and testing whether a 1.5B model still fits the budget once the hybrid gate is doing this much of the work.

Built With

• aarch64
• arm
• cortex-a76
• edge-ai
• gguf
• iot
• llama.cpp
• neon
• numpy
• pandas
• pytest
• python
• quantization
• qwen2
• raspberry-pi