FastPath64: the missing Arm smmla kernel for IQ4_XS

IQ4_XS is the smallest standard 4-bit GGUF and had a tiled matmul kernel on Intel AMX, none on Arm. FastPath64 writes it: 2.12x prefill on Neoverse N2, 1.30x a whole agent turn, byte-identical output.

Comment

FastPath64 — Track 2, Cloud AI

Project Overview

The format you pick to fit an Arm instance was the one Arm could not accelerate

Every 4-bit GGUF makes the same two decisions: which 16 values a nibble may take, and how finely the scale that rescales them may vary. IQ4_XS is the only widely published format that takes the good answer to both.

nibble values scale structure bits/weight
Q4_0 uniform grid one fp16 per 32 weights 4.50
Q4_K uniform grid super-block + 6-bit scale and 6-bit min per 32 4.50
IQ4_NL non-uniform codebook one fp16 per 32 weights 4.50
IQ4_XS non-uniform codebook super-block + 6-bit scale per 32, no min 4.25

The codebook matters because weights are roughly normally distributed, and a uniform grid spends half its levels on a tail that is nearly empty; kvalues_iq4nl places its 16 levels where the mass actually is. The super-block matters because one fp16 per 32 weights is 0.5 bpw of pure metadata. IQ4_XS takes the codebook from the IQ family and the scale hierarchy from the K family, and by dropping the min term it lands cheaper than either. Each figure is one struct definition away from being checked: 136/256, 144/256, 18/32.

That ranking is what actually gets published. In the standard GGUF release ladder — 18 files, IQ3_M through f16 — IQ4_XS is the only IQ4 variant shipped at all, and the smallest 4-bit rung:

IQ3_M · IQ4_XS · Q3_K_L · Q3_K_XL · Q4_0 · Q4_0_4_4 · Q4_0_4_8 · Q4_0_8_8
Q4_K_S · Q4_K_M · Q4_K_L · Q5_K_S · Q5_K_M · Q5_K_L · Q6_K · Q6_K_L · Q8_0 · f16

Arm's fast path accelerated IQ4_NL — the 4.5 bpw cousin that ladder does not contain — and not IQ4_XS. The one IQ format on Arm's fast path is the one nobody downloads. Meanwhile the three Q4_0_4_* files above are the tail of an earlier answer to this same problem: Arm-specific pre-repacked GGUFs, which upstream removed in favour of doing the interleave at load time (ggml/src/ggml.c:894 — "TYPE_Q4_0_4_4 REMOVED, use Q4_0 with runtime repacking") so nobody would have to publish an Arm-shaped file again. That runtime mechanism is exactly the one IQ4_XS was never added to.

Two doors, both shut

llama.cpp reaches Arm's integer matrix-multiply unit through two mechanisms. The first is KleidiAI, Arm's own microkernel library, which selects between DOTPROD, I8MM, SVE and SME2 kernels at runtime. The second is the in-tree repack path, which converts weight tensors into a row-interleaved layout and issues smmla — the Armv8.6 8-bit integer matrix-multiply-accumulate instruction — over 2×8 operand tiles.

Neither mechanism accepted IQ4_XS. KleidiAI dispatches only on GGML_OP_MUL_MAT and only for Q4_0 and Q8_0 (kleidiai.cpp:536,668), which also excludes GGML_OP_MUL_MAT_ID, the operation through which every mixture-of-experts expert matmul is routed. The repack type table (ggml_repack_get_optimal_repack_type, repack.cpp:4528) enumerates Q4_0, Q2_K, Q4_K, Q5_K, Q6_K, Q8_0, IQ4_NL and MXFP4. IQ4_XS occurs zero times in the file.

On Neoverse hardware those weights fell through to the generic per-row vec_dot path: sdot at best, smmla never issued, on a core whose lscpu advertises i8mm. The same format is enumerated in qtype_has_amx_kernels() (amx/common.h:106), so Intel AMX carries a tiled path for it. The gap was specific to Arm.

So the incentive closes on itself. Selecting an Arm instance for cost efficiency motivates fitting the largest model into the least memory, which motivates the most aggressive 4-bit format available — which was precisely the format with no Arm kernel. The optimisation chosen to fit the instance surrendered the throughput it was intended to buy.

The trade-off, priced

From the benchmark run's own metadata, on one 4 vCPU Neoverse N2 instance:

Llama-3.2-3B file on disk prefill pp512, stock prefill, FastPath64
Q4_K_M 1.873 GiB 42.70 t/s 42.74 t/s (untouched control)
IQ4_XS 1.696 GiB (−9.5%) 24.95 t/s (0.58×) 52.85 t/s (1.24×)

Before this work, choosing IQ4_XS bought 9.5% less memory for 42% less prefill throughput. Prefill throughput is what sets how many agent turns an Arm instance absorbs per second, so the format chosen to make the model fit cost roughly 1.7× the instances to serve it.

After this work there is no trade at all. IQ4_XS is smaller and faster than Q4_K, on the same unmodified file, on the same hardware. (OLMoE-1B-7B: 3.458 vs 3.922 GiB, −11.8%.)

Smaller and faster

And the two properties are the same property. IQ4_XS is small because it has no min term; the kernel is fast because it has no min term — a single 6-bit scale covers both nibble halves of a sub-block, so no bsums/dmin correction is needed and the inner loop does strictly less arithmetic per byte than Q4_K's. The format that was the slowest 4-bit option on Arm was structurally the one that should have been the fastest. It was losing to a missing kernel, not to its own design.

Result

Prefill throughput for Llama-3.2-3B in IQ4_XS on Neoverse N2 rises 2.12× at 512-token prompts and 1.60× at 2048, on an unmodified GGUF file, with Q4_K — a format the change does not touch — held as a control at 1.00×. A whole agent turn runs 1.30× faster end to end.

Prefill speedup

Functionality / Output

Why the kernel did not previously exist

IQ4_XS is structurally a hybrid. Its quantised values are indices into kvalues_iq4nl, the same 16-entry non-linear codebook used by IQ4_NL, while its scale structure is K-quant style: a super-block d in fp16 plus eight 6-bit sub-block scales split across scales_h (2 bits) and scales_l (4 bits), each covering 32 elements.

typedef struct {
    ggml_half d;                  // super-block scale
    uint16_t  scales_h;           // 2 high bits x 8 sub-blocks
    uint8_t   scales_l[QK_K/64];  // 4 low bits  x 8 sub-blocks
    uint8_t   qs[QK_K/2];         // codebook indices
} block_iq4_xs;

Arm already had interleaved smmla kernels for each half separately — block_iq4_nlx8 handles the codebook, block_q4_Kx8 handles super-block scales against q8_K activations. The missing kernel was the intersection of two kernels already in the tree, which is a plausible explanation for why it was never written: it belonged to neither the IQ-quant nor the K-quant effort.

Layout

block_iq4_xsx8 interleaves eight rows, taking eight bytes at a time round-robin, so row j's byte (8c + i) is placed at qs[64c + 8j + i]. The interleaved block is exactly 8 × sizeof(block_iq4_xs) = 1088 bytes, a hard constraint: the repack buffer type allocates ggml_nbytes(tensor), leaving no room to pre-decode scales into a friendlier form.

The consequence is that eight consecutive bytes belong to a single column, so one 16-byte load yields a column pair — precisely an smmla operand.

Kernels

vmmlaq_s32(acc, a, b) treats both operands as 2×8 int8 matrices and returns the 2×2 product. With weights as one operand and activations as the other, a 16-byte weight load meets a 16-byte activation load with no shuffle in the inner loop:

const int8x16_t w[4] = {                              // 4 chunks -> q8 groups 0..3
    vqtbl1q_s8(kvalues, vandq_u8 (raw0, m4b)),        // local elements  0-7
    vqtbl1q_s8(kvalues, vandq_u8 (raw1, m4b)),        //                 8-15
    vqtbl1q_s8(kvalues, vshrq_n_u8(raw0, 4)),         //                16-23
    vqtbl1q_s8(kvalues, vshrq_n_u8(raw1, 4)),         //                24-31
};
for (int t = 0; t < 4; t++) {
    sb01 = vmmlaq_s32(sb01, w[t], q8_01[t]);          // activation rows 0,1
    sb23 = vmmlaq_s32(sb23, w[t], q8_23[t]);          // activation rows 2,3
}

Result lanes are [w0·a0, w0·a1, w1·a0, w1·a1], so lanes 0–1 carry the first column's scale and lanes 2–3 the second's, and the scale vector is built as [s0, s0, s1, s1].

Three kernels ship, covering the deployed Arm server population:

path instruction hardware
GEMM smmla (I8MM) Graviton3/4, Cobalt 100, Axion
GEMM sdot (DOTPROD) Graviton2, Ampere Altra
GEMV sdot (DOTPROD) all of the above

GEMV remains on DOTPROD by design. At nr == 1 there is no second activation row with which to fill an smmla operand, and the existing non-repacked vec_dot path is already well optimised; displacing it would trade batch-1 decode for prefill. Activations are instead loaded once per 8-element run and duplicated so that a single vdotq_s32 serves both columns of a pair, keeping all sixteen of its multiply-accumulates productive.

Dispatch is gated at DOTPROD. Below that threshold the portable reference is slower than not repacking at all, so IQ4_XS is deliberately left on the existing path.

Evaluation

Both arms are built from the same pinned upstream commit in the same CI job on the same physical runner, eliminating machine-to-machine variance. Timings are gated behind a numerical equivalence check and a GGUF provenance check that run first; the workflow refuses to report throughput if either fails.

Llama-3.2-3B, IQ4_XS stock FastPath64
prefill pp512 24.95 ±0.02 t/s 52.85 ±0.02 t/s — 2.12×
prefill pp2048 17.72 ±0.01 t/s 28.31 ±0.01 t/s — 1.60×
Q4_K control 42.70 ±0.06 t/s 42.74 ±0.04 t/s — 1.00×

The gain is a property of the kernel rather than of one model size. On gemma-4-12b-it-IQ4_XS, four times larger and benchmarked alone on its own runner, prefill rises 2.00× at pp512 and 1.61× at pp2048 — tracking the 3B figures closely. Three independent runs on separate runner instances returned these ratios to two decimal places.

Attribution rests on the control rather than on the delta alone. Q4_K is untouched by this change and lands at 1.00× to two decimal places across all four cases; had the patched build benefited from an incidental effect — compiler behaviour, cache state, a quieter neighbour — the control would have moved with it.

The mechanism was isolated before any kernel was written, by toggling GGML_CPU_REPACK across three builds of the same commit:

Repack toggle

Disabling Arm's fast path costs Q4_K 35% of its prefill throughput and costs IQ4_XS 0.6%, inside the run-to-run spread. The absence was demonstrated rather than inferred: a format cannot lose what it never had.

What it is worth on a real agent turn

llama-bench measures a kernel. An agent turn measures what the instance is rented for: a large system prompt, tool schemas, history and retrieved context in; a short structured tool call out. Measured end to end on the same runner — 5145 prompt tokens, 96 generated, median of 3:

stock FastPath64
time to first token 463.48 s 355.13 s — 1.31×
decode 6.59 s 6.55 s — 1.01×
whole turn 470.09 s 362.28 s — 1.30×

99% of the stock turn is prefill, which is why a prefill kernel moves the number a user feels: 108 seconds of every 470-second turn, given back on an unmodified file. Absolute times are a property of a free 4 vCPU runner; the ratio is the result.

On the same prompt with greedy decode and a fixed seed, the two builds emit byte-identical output — the same tool call, token for token. Both checks run in the same workflow as the throughput numbers.

The speedup declines with prompt length, and the trend is the argument's own consistency check:

prompt tokens 512 2048 5145
speedup 2.12× 1.60× 1.31×

This work touches the FFN matmuls and nothing else. Attention grows with sequence length, so the share of prefill available for a matmul kernel to improve shrinks. No headline figure here is quoted without its prompt length.

The MoE measurement is lower than the dense one (1.13× on OLMoE-1B-7B) for a reason visible in the same arithmetic: a MoE prefill reads every expert's weights while computing only the active fraction, so its arithmetic intensity is lower by roughly the sparsity ratio, moving it toward the bandwidth-bound regime in which a compute kernel cannot help.

Correctness

Repacked kernels are unreachable from test-backend-ops, which allocates tensors into the default CPU buffer; the repack path activates only for tensors resident in the repack buffer type. This submission therefore includes an equivalence harness that allocates identical weights into both buffer types and diffs the results.

On Neoverse N2, before any timing is reported:

type=iq4_xs N=64  K=512  M=1   max_abs=3.815e-06  PASS   (GEMV only)
type=iq4_xs N=64  K=512  M=5   max_abs=3.815e-06  PASS   (both paths + remainder)
type=iq4_xs N=128 K=1024 M=9   max_abs=7.629e-06  PASS
type=iq4_xs N=8   K=256  M=16  max_abs=0.000e+00  PASS   (GEMM only)

Worst-case deviation is below that of both the portable reference and upstream's own Q4_K repack kernel, smmla accumulating exactly in int32 with fewer intermediate float roundings. The harness also surfaced a latent fault upstream: init_tensor leaves tensor->extra null for any type no kernel claims, and set_tensor dereferenced it unchecked, converting a configuration error into a segmentation fault. A diagnostic assertion is included in the patch series.

Correctness is further verified across dispatch paths no free CI runner exposes, by cross-compiling at native speed and executing under QEMU with a selected core model — neoverse-n1 reproduces the DOTPROD-only behaviour of Ampere Altra and Graviton2. Twenty-four combinations pass: six shapes × {IQ4_XS, Q4_K} × {N2, N1}. Emulation is used for correctness only; no timing taken under it is reported.

Artifacts

  • A three-patch series against pinned upstream llama.cpp, on a branch prepared for review with a written PR description.
  • test_repack_equiv.cpp, an equivalence gate for repacked kernels covering a case upstream's test suite structurally cannot reach.
  • A QEMU cross-build harness giving access to DOTPROD-only and I8MM dispatch paths without Arm hardware or a cloud account.
  • agentbench.py and diff_generation.sh, measuring a whole tool-calling turn and checking that the output text does not change.
  • gguf_types.py, a dependency-free GGUF tensor-type parser — because a file named IQ4_XS need not contain any, and benchmarking one that does not returns a clean 1.00× that looks like a result.
  • docs/the-gap.md, a source-level audit with file:line citations of what each Arm fast path accepts.
  • Benchmark workflows any reader can execute on free hardware.

Limitations

Decode is not improved: batch-1 decode is memory-bandwidth-bound and no matmul kernel alters that. It carries a small regression (0.97× on the 3B, 0.92× at 12B) from per-sub-block scale decoding the non-repacked path does not perform. One fix was written, measured, and rejected — decoding the scales with vld4_u8 cost ~10% of prefill because the vector result was consumed as eight scalars and stalled on store-to-load forwarding; the mechanism is recorded in the repository. Output is numerically equivalent rather than guaranteed bit-identical, smmla accumulating in a different order than the reference path. The AMX comparison is read from upstream source rather than measured. KleidiAI is untouched: extending Arm's own library to MUL_MAT_ID and the IQ formats is the larger fix, and it belongs upstream of llama.cpp rather than in a patch against it.

Setup Instructions

Three routes, in increasing order of what you need to have. None requires paying for Arm hardware.

1. Benchmarks on Arm, without an account or spend. In the repository, open Actions → A/B - stock vs FastPath64 on Neoverse N2 → Run workflow. The job clones the pinned upstream twice, patches one copy, builds both on the same free Neoverse N2 runner, runs the equivalence and provenance gates, and reports throughput only if they pass. Runtime is 1h30–2h30, most of it building llama.cpp twice on 4 vCPUs; for a ~40 minute sanity run set repeats: 2 and prompt_sizes: 512 in the dispatch form. Expected output is a step-summary table in which the IQ4_XS rows move ~2.12× and the Q4_K control rows read 1.00×.

2. Correctness locally, on x86, without Arm hardware. Cross-compiles at native speed and executes under emulation across both dispatch paths. Needs Docker, 10–20 minutes:

git clone https://github.com/Marc-Dvci/fastpath64 && cd fastpath64
docker build -t fastpath64-cross tools/qemu/
git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git ../upstream-llama.cpp
git -C ../upstream-llama.cpp checkout "$(cat UPSTREAM_SHA)"
for p in patches/*.patch; do git -C ../upstream-llama.cpp apply "$p"; done
docker run --rm -v "$PWD/..:/src" fastpath64-cross bash /src/fastpath64/tools/qemu/run-equiv-test.sh

Expected: 24 PASS lines, each preceded by took repack fast path: yes, then ALL CHECKS PASSED.

3. On your own Arm64 server — Graviton, Axion, Cobalt, Ampere Altra. No flags are required; kernel selection follows runtime CPU feature detection, so one binary picks smmla on an I8MM core and sdot on one without. Dependencies: cmake, a C++17 compiler, curl, python3.

git clone https://github.com/Marc-Dvci/fastpath64 && cd fastpath64
git clone --filter=blob:none https://github.com/ggml-org/llama.cpp.git upstream
git -C upstream checkout "$(cat UPSTREAM_SHA)"
for p in patches/*.patch; do git -C upstream apply "$p"; done

cmake -S upstream -B upstream/build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build upstream/build --target llama-bench -j"$(nproc)"

curl -fL -o m.gguf https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-IQ4_XS.gguf
upstream/build/bin/llama-bench -m m.gguf -p 512,2048 -n 128 -t "$(nproc)" -r 5

Build a second tree from the same commit without applying the patches to see the delta; bench/compare_ab.py formats the comparison and bench/gguf_types.py confirms the file really is IQ4_XS before you trust either. If both builds give the same pp512 number on a core whose /proc/cpuinfo lists i8mm, the patches did not apply.

Built With

  • aarch64
  • arm
  • arm-neon
  • c
  • c++
  • cmake
  • ggml
  • gguf
  • github-actions
  • i8mm
  • iq4-xs
  • kleidiai
  • llama.cpp
  • neon-intrinsics
  • neoverse
  • python
  • qemu
  • quantization
  • smmla
  • sve2
Share this project:

Updates