Arm Edge AI: The Quantization Trap and the Fix

Naive Quantization Is a Trap: The Real Way to Speed Up Edge AI on Arm

Track: Physical AI / Edge AI — running entirely on a Raspberry Pi 5 (Cortex-A76, aarch64).

Inspiration

Every "optimize for edge AI" guide says the same thing: quantize to INT8, get a free speedup. We wanted to actually measure that claim on real Arm hardware instead of taking it on faith — and found a trap that most tutorials never mention.

What it does

Two honest, reproducible benchmarks on a Raspberry Pi 5:

1. The trap: naive INT8 quantization (manual scale/round/clip) multiplied via NumPy's default int32 matmul. Result: 100-300x SLOWER than plain FP32, because NumPy's integer matmul has no vectorized Arm NEON kernel — only its float BLAS path does.
2. The fix: the same quantization idea, but run through ONNX Runtime (which dispatches to genuine Arm-optimized MLAS/XNNPACK kernels using the asimddp NEON dot-product instruction present on this CPU). Result: up to 6.18x FASTER at batch=1, plus a 4x smaller model (9.5MB -> 1.2MB).

How we built it

• benchmark.py: manual INT8 quantization vs FP32, pure NumPy, isolates the "trap".
• build_model.py: builds a representative 4-layer edge-classifier MLP (784->512x4->10) directly as an ONNX graph.
• Dynamic INT8 quantization via onnxruntime.quantization.quantize_dynamic.
• onnx_benchmark.py: FP32 vs INT8 inference latency through ONNX Runtime's CPU execution provider, across batch sizes 1/8/32, isolates the "fix".

Challenges we faced

The counterintuitive result (naive INT8 being slower) was surprising enough that we double- and triple-checked it wasn't a benchmarking mistake before trusting it. It turned out to be real and reproducible: NumPy's BLAS backend (OpenBLAS) is NEON-optimized for float matmul, but its generic integer matmul path is not vectorized at all.

What we learned

Quantization is not automatically a win on Arm — it depends entirely on whether your runtime has a real Arm-tuned integer kernel. The speedup is largest exactly where it matters most for edge deployment: single-sample, low-latency inference (batch=1: 6.18x), and shrinks toward memory-bandwidth-bound territory at larger batches (batch=32: 1.16x).

What's next

Extending the benchmark to a real quantization-aware-trained model (not just post-training dynamic quantization) and to Arm's KleidiAI kernels directly, to see how much further the batch=1 latency can drop for genuinely resource-constrained physical AI deployments.

Full code, raw JSON results, and reproduction steps: see the GitHub repo linked above.

Built With

• arm
• edge-ai
• inference-optimization
• neon
• onnx
• quantization
• raspberry-pi