kleidi-advisor

• 

  Every Q4_K_M audited misses Arm's KleidiAI kernels. Measured cost: 1.61× prompt throughput at +0.049 perplexity on Cobalt 100.

• 

  The tool's own output, before and after. One imatrix-calibrated requant moves the model from NOT_KLEIDIAI_PATH to OK_KLEIDIAI.

• 

  Which buffer the tensors land in is the only reliable signal — and it prints only under -v. Grepping for "repack" or "kleidi" is unreliable.

• 

  211.8 MB of HTTP range requests classified 17 published GGUFs without downloading one. 13 never reach KleidiAI's kernels.

• 

  The original premise was wrong and my own verify would have confirmed it. Caught by reading raw load logs as QA. see README §8.

What I built

kleidi-advisor is a CLI — scan, audit, fix, bench, report — that answers a question nothing else in the llama.cpp ecosystem answers:

will this GGUF actually reach Arm's KleidiAI kernels, and what does it cost me if it doesn't?

Track 2 — Cloud AI. All measurements on Azure Standard_E8ps_v6 (Azure Cobalt 100, Arm Neoverse-N2), 8 vCPU, Ubuntu 24.04 arm64, llama.cpp b10431.

The finding

All 7 Q4_K_M files in my audit list miss KleidiAI's kernels — every one, across all 5 publishers that ship one: bartowski, lmstudio-community, TheBloke, unsloth, and Qwen's own repo. In total, 13 of 17 successfully scanned GGUFs never reach those kernels. Classifying all 17 cost 211.8 MB of traffic, because scan reads the GGUF header over an HTTP range request and stops — no model download required.

Measured on Qwen2.5-7B-Instruct:

1.61× prompt processing at +0.049 perplexity — a 0.6% quality cost that sits inside the error bars of both measurements. That is a statement about resolution, not equivalence.

The speedup belongs to Arm's KleidiAI kernels. What I built detects the miss and measures what it costs.

What inspired it

llama.cpp used to ship pre-packed Arm formats (Q4_0_4_4, Q4_0_8_8) where the fast path was visible in the filename. Upstream replaced them with runtime repacking of plain Q4_0 — better for the ecosystem, but it made kernel dispatch invisible in the file. Meanwhile Q4_K_M became the community default. Nobody re-checked what that combination costs on Arm.

What I learned

I built this expecting two outcomes: Q4_0 reaches Arm's kernels, K-quants fall back to generic code, with the published ~2.5–2.9× uplift from llama.cpp PR #9921 as the target.

The box falsified that, and the advisor tool nearly hid it

On b10431 there are three paths. Q4_K_M isn't unaccelerated — it lands in ggml's own aarch64 repack buffer. The real gap is optimised-vs-differently-optimised: 1.61×, not 2.5–2.9×.

Worse: my --verify matched substrings repack, kleidi, aarch64. On this build all three appear in both formats' logs, so the pattern-hit branch always fired and the outcome was decided entirely by the verdict under test. A step whose purpose was to falsify the table would have rubber-stamped it. It was caught only by reading raw -v load logs side by side before trusting my own output.

The fix: buffer-type detection instead of substring matching. The signal that separates the paths is which model buffer tensors land in, and it only prints under -v:

Q4_0:   load_tensors: CPU_KLEIDIAI model buffer size = 3500.45 MiB
Q4_K_M: load_tensors:   CPU_REPACK  model buffer size = 4166.82 MiB

That whole episode is documented in README §8. A classification table never contradicted is a table never tested.

How I built it

Spec-first: a full implementation plan and a byte-level reference (GGUF v3 layout, ggml type IDs, expected tool output shapes) written before any code, so the agent building it had no reason to guess a magic number. The CLI was then built and verified entirely offline — GGUF fixtures generated by a test-local writer, llama.cpp exercised through stub binaries, HTTP range requests against an in-process fixture server. 118 tests pass without network, models, or an Arm CPU.

scan --url works because GGUF puts all metadata before the tensor data: fetch the first 2 MiB and you can classify a 15 GB model.

Challenges

• llama.cpp's warning is real but insufficient. It does log kleidiai: no kernel for tensor type q4_K — at load time, after you've downloaded several GB, amid the KV dump, quantifying nothing. "Not accelerated" doesn't tell you if it's 2% or 2×.
• The obvious grep is a trap. Both formats print cannot be used with preferred buffer type CPU_KLEIDIAI — a line containing "KLEIDIAI" that means the opposite.
• llama-cli goes interactive with no prompt and hangs a verification step forever.
• 8 vCPU budget. Full-corpus perplexity is hours, so both sides run --chunks 100 — disclosed, identical, and the absolute PPL is explicitly not comparable to published figures.

Limitations, stated up front

One machine, one model, one build, one corpus sample. Only Q4_0 and Q4_K_M had buffers observed directly; other K-quants are classified by family and no IQ model was loaded.

The audit list is hand-assembled, not download-ranked. Every one of those is one scan --verify away from being falsified, which I guess is the point.

Built With

• aarch64
• arm
• azure
• benchmarking
• claude-code
• cobalt-100
• cpu-inference
• gguf
• imatrix
• kleidiai
• llama.cpp
• llm
• neoverse
• perplexity
• pytest
• python
• quantization
• qwen2.5