arm-inference-audit

What it is

arm-inference-audit checks whether a llama.cpp build on Arm is actually using KleidiAI kernels, and measures the performance and cost impact on Google Axion (Neoverse-V2).

Inspiration

On Arm cloud CPUs, llama.cpp can look fine at runtime (backend: CPU) while the binary contains zero KleidiAI matmul symbols. Enabling GGML_CPU_KLEIDIAI=ON compiles real kernels, but normal output still does not say which path is active. That invisible dispatch gap is what this project targets.

What I built

• scripts/audit.sh — compares claims (runtime banner) vs contains (nm symbol counts)
• Locked llama-bench scripts for base vs KleidiAI builds
• Measurements on GCP c4a-standard-8 (Axion), commit 9558fa44c
• Results for 0.5B and 7B, weight quant, and KV-cache quant

What I learned

• This is not “accelerated vs unaccelerated”: both builds use Arm i8mm / dotprod / sve. The comparison is KleidiAI Q4_0 kernels vs ggml’s native Arm kernels.
• Upstream issue #26630 (zero kernels under the documented build line) did not reproduce on Neoverse-V2 here. CMake auto-detect worked; the real variable was simply KleidiAI on vs off.
• KleidiAI gave a small prefill gain and a small decode loss. KV-cache quantization (q8_0 / q4_0) gave a much larger prefill boost (~28–30%) with almost no decode penalty — the stronger cost lever on the same box.

Challenges

• Runtime is silent: you cannot tell dispatch from logs alone; you need binary inspection.
• Early hypothesis (manual GGML_CPU_ARM_ARCH workaround) was wrong on this platform; had to pivot without wasting the deadline.
• Short perf sampling did not give stable “executes” evidence, so the verifier ships claims + contains only, honestly.

Why it should win

It is a reusable developer tool for Arm CPU inference: one command to verify KleidiAI dispatch, plus measured cost framing on real Axion hardware — not just another benchmark screenshot.

Built With

• arm
• axion
• bash
• cmake
• gcp
• google
• kleidiai
• llama-bench
• llama.cpp
• neoverse-v2