Arm ISA Dispatch Audit: SVE2 advertised, never used

Project Overview

Track: Cloud AI. An ISA-level dispatch audit for llama.cpp + KleidiAI on Neoverse server cores (Graviton / Cobalt / Axion class), reproducible on free Arm CI.

Neoverse N2 and Azure Cobalt 100 advertise sve sve2 svei8mm svebf16. Read /proc/cpuinfo on one and you will believe the SVE GEMM kernels are doing your matrix multiplies.

They are not.

KleidiAI gates every SVE kernel behind a runtime check for a 256-bit vector:

// llama.cpp - ggml/src/ggml-cpu/kleidiai/kleidiai.cpp
if (svcntb() == 32) { features |= CPU_FEATURE_SVE; }

svcntb() returns the SVE vector length in bytes, so 32 means 256-bit. N2 and Cobalt are 128-bit parts. svcntb() returns 16, CPU_FEATURE_SVE is never set, and every *_sve_* kernel in the dispatch table is unreachable dead code on the most widely-available Neoverse silicon - including the free GitHub Actions runner every developer can reach.

Measured on ubuntu-24.04-arm, not asserted:

SVE present : yes
svcntb()    : 16 bytes (128-bit vector)

FEATURE              ADVERTISED   KLEIDIAI     NOTE
dotprod (i8 dot)     yes          USES         NEON path, always reachable
i8mm    (int8 mm)    yes          USES         neon_i8mm kernels - the real path
bf16                 yes          USES         bf16 kernels where present
SVE / SVE2           yes          refuses      ADVERTISED BUT UNREACHABLE - kleidiai.cpp:209 needs svcntb()==32

=== VERDICT ===
This core advertises SVE, but KleidiAI will NOT dispatch a single SVE kernel.
Required: svcntb() == 32 (256-bit).  Actual: 16 (128-bit).
Every *_sve_* GEMM kernel in the dispatch table is dead code here.
Your matmul is running the neon_i8mm path.

Why it should win

Every "deploy an LLM on Arm with llama.cpp and KleidiAI" guide tells you to check /proc/cpuinfo for SVE2. On the server parts most developers can actually rent, and on the CI runner they get for free, that flag describes a code path the process will never enter. The optimization you believe you enabled is not running, and nothing warns you.

This entry is not a wall-clock speedup claim. It is an ISA-level dispatch audit with a root cause and an exact source line, reproducible by a judge on their own free runner in about two minutes without hardware or cloud spend. It reverses an assumption the whole field is operating under, and it is checkable in 30 seconds by anyone who knows the code.

Honesty guardrails

An Arm engineer will check these, so they are stated up front rather than buried:

• The gate is KleidiAI's, not ggml's. ggml proper does use SVE elsewhere (vec.cpp, simd-gemm.h). The claim is scoped to KleidiAI's kernel dispatch. "ggml ignores SVE" would be false.
• The SVE kernels exist. There is a ..._16x8_sve_i8mm GEMM kernel in the table. The claim is that it is unreachable on 128-bit cores, not that it is absent.
• The portable build refuses to guess. svcntb() is an SVE intrinsic, so a build without +sve cannot read the vector length. Rather than print a fabricated 0, it reports unknown, prints the rebuild command, and exits 3. Only the +sve build renders a verdict. This was a real bug caught by CI on the first run, when the portable build printed Actual: 0 (0-bit) - a number it had not measured.
• Apple Silicon has no SVE at all. On an M2 the probe exits 0 with "the NEON/i8mm path is the only path", which is why benchmark numbers from SVE-capable hardware transfer nowhere.

What this optimizes, and the before/after

The reminder asks what was optimized. This project's target is the developer's optimization decision itself - the step before any latency number is meaningful.

Before: you read /proc/cpuinfo, see sve sve2 svei8mm svebf16, follow an Arm Learning Path, and benchmark. You attribute your throughput to the SVE i8mm kernels. Any tuning you do from there is aimed at a code path your process never enters.

After (measured, ubuntu-24.04-arm, run 30711783573):

	Before (assumed)	After (measured)
Kernel family in use	*_sve_i8mm GEMM	neon_i8mm GEMM
SVE vector length	assumed >= 256-bit	svcntb() = 16 bytes = 128-bit
CPU_FEATURE_SVE	assumed set	never set
SVE kernels reachable	assumed yes	zero of them
How you find out	you do not	./isa-probe-sve; echo $? -> 2

The delta this produces is not a percentage on a token rate - it is that every SVE-targeted optimization on 128-bit Neoverse has an expected speedup of exactly zero, and this tells you so in two minutes instead of after a week of tuning. On hardware that is 256-bit, the same probe exits 0 and your SVE work is real.

Cross-checked on Apple M2 (no SVE at all, exits 0) so the comparison is honest about which ISA did what.

Functionality / Output

isa-probe prints what the CPU advertises next to what KleidiAI will actually dispatch, then renders a verdict with a distinct exit code so it drops straight into CI as a gate:

Exit	Meaning
0	No SVE on this core, or SVE present and 256-bit (KleidiAI would use it)
2	Advertised but refused - the core has SVE, KleidiAI will not dispatch it
3	Advertised, but this build cannot measure the vector length (rebuild with +sve)

The GitHub Actions workflow additionally:

• prints the runner's own /proc/cpuinfo feature line, core count and memory, so the hardware is on the record
• builds both the portable and the +sve variant, and records it when the toolchain cannot target SVE rather than hiding it
• clones upstream llama.cpp and greps svcntb out of the real source, so the 256-bit threshold is read from the code rather than taken from this description - if upstream changes the constant, the step surfaces it
• enables PMU counters (kernel.perf_event_paranoid=-1) and runs perf stat to confirm real hardware counters are live
• uploads both binaries as an artifact

Setup Instructions

Reproduce on a free Neoverse runner (no hardware, no cloud spend)

1. Fork https://github.com/luongs3/arm-dispatch-audit
2. Actions tab, select arm-dispatch-audit, click Run workflow
3. Open the step named "The finding - advertised vs dispatched"

The runner is ubuntu-24.04-arm (Neoverse N2 / Azure Cobalt 100), free on public repositories, 4 vCPU / 15 GB. Expect the +sve build to report svcntb() : 16 bytes (128-bit vector) and exit 2.

Reproduce locally on any Arm machine

git clone https://github.com/luongs3/arm-dispatch-audit
cd arm-dispatch-audit

# The build that can actually answer the question:
cc -O2 -march=armv8.2-a+sve -DHAVE_SVE -o isa-probe-sve src/isa-probe.c
./isa-probe-sve; echo "exit: $?"

# The portable build (works anywhere, declines to guess about SVE):
cc -O2 -o isa-probe-portable src/isa-probe.c
./isa-probe-portable; echo "exit: $?"

No dependencies beyond a C compiler. On Apple Silicon the +sve build is unnecessary - there is no SVE to measure - and the probe will say so.

Validate the claim against upstream yourself

git clone --depth 1 https://github.com/ggml-org/llama.cpp
grep -rn "svcntb" llama.cpp --include=*.cpp --include=*.h

You should find the svcntb() == 32 comparison that gates CPU_FEATURE_SVE.

Repository

https://github.com/luongs3/arm-dispatch-audit (public, Apache-2.0)

Built With

• aarch64
• arm
• c
• ggml
• github-actions
• kleidiai
• llama.cpp
• neoverse
• perf
• simd
• sve
• sve2