ArmForge

Inspiration

Most on-device optimization write-ups stop at "I quantized a model and it got faster." That is true, but it hides the mechanism, and the mechanism is what tells you what to optimize next. I wanted a small, reproducible harness that measures what actually makes an LLM fast on Arm silicon, ties each speedup to a specific cause, and stays honest when a popular trick turns out to do nothing.

What it does

ArmForge takes one real model (Qwen2.5-1.5B-Instruct) and one Arm CPU (an Apple M4, no GPU, no cloud) and produces four measured findings, each with a chart and a one-command reproduction. Everything runs on the CPU only (llama.cpp, Metal off), so the numbers reflect the Arm cores themselves.

1. Build it right for Arm (about 5x). The most common trap on Apple Silicon: a Homebrew cmake running under Rosetta reports x86, so llama.cpp compiles zero Arm SIMD kernels and falls back to generic C. A native arm64 build is about 5x faster on the same source, same model, and same chip.

2. Generation is bandwidth-bound, so quantization is the lever. Generation throughput tracks bytes moved, not FLOPs. From fp16 to Q8_0 to Q4_0 it gets faster as the weights get smaller (about 3.5x from fp16 to Q4_0), at up to 3.3x smaller on disk.

3. The Arm int8 ISA (dotprod / i8mm) does not speed up generation. Rebuild the same weights and the same C++ against three Arm arch levels (armv8-a, +dotprod, +i8mm) and generation barely moves. Single-token generation is a matrix-vector product, which is bandwidth-bound; i8mm only speeds up matrix-matrix products, which is prompt processing. So it cannot help the generation path.

4. Quality, measured honestly, including a null result. Q8_0 is effectively lossless (perplexity 9.157 vs fp16 9.161). Q4_0 costs a small, quantified bump (9.623). An importance matrix barely moves legacy Q4_0 (9.623 to 9.623), because imatrix pays off on k-quants and sub-4-bit formats, not on round-to-nearest Q4_0. The null result is reported rather than hidden.

It ships as one command (setup.sh then run_all.sh) that emits the CSVs, the four charts, and a summary, plus a fully local CPU chat demo that answers at about 98 tokens/sec.

How we built it

• Engine: llama.cpp, CPU backend, Metal off, built with a native arm64 cmake (this is finding 1).
• ISA control: -DGGML_CPU_ARM_ARCH=armv8.2-a+fp16[+dotprod[+i8mm]] to toggle Arm int8 instructions at build time.
• Quantization: llama-quantize for Q8_0 and Q4_0, plus llama-imatrix calibrated on wikitext-2.
• Measurement: llama-bench for throughput, thread scaling, and the build comparison; llama-perplexity for quality; Python and matplotlib for the charts.
• Hardware: Apple M4, 4 performance and 6 efficiency cores, with DotProd, i8mm, BF16, and SME2 present.

Challenges we ran into

• The toolchain itself was the trap, and it became finding 1. Diagnosing "zero Arm SIMD kernels compiled" under a Rosetta cmake took real digging: x86 flags, a cross-compile ABI failure, and an unrelated OpenSSL/httplib link error. The fix is a native arm64 cmake plus explicit -march and -DCMAKE_DISABLE_FIND_PACKAGE_OpenSSL=ON.
• The first hypotheses were wrong, and the data said so. A mis-built baseline made it look like "Q8 beats Q4" and "i8mm gives 2x." Correctly built binaries reversed both. The narrative was rebuilt around what the clean numbers actually show, which turned out to be a better, mechanistic story.
• KleidiAI (including SME2) fails to build on current llama.cpp master because of an upstream version skew. It is documented as a frontier rather than shipped.

Accomplishments that we're proud of

A rigorous, reproducible harness that attributes on-device speed to causes, and is willing to publish null results (imatrix on Q4_0, i8mm on generation) instead of overselling.

What we learned

On Arm, on-device generation is bound by memory bandwidth, not compute. The things to optimize are bytes moved (quantization) and the kernel and toolchain. Do not expect newer int8 math instructions to help the generation path.

What's next for ArmForge

Energy per token for a battery angle, Q4_K with an importance matrix where it actually helps, prompt-processing benchmarks where i8mm should pay off, and a port to AWS Graviton for a cloud cost-per-token story.

Built with

llama.cpp, ggml, Arm NEON / dotprod / i8mm, Qwen2.5, Apple M4, Python, matplotlib

Built With

• apple-m4
• arm
• cpp
• dotprod
• ggml
• i8mm
• llama.cpp
• matplotlib
• neon
• python
• qwen2.5