Whetstone

Arm's recommended fast path triples generation latency on an M1 Max and speeds up two cloud chips. Whetstone is the instrument that catches it

Comment

Try it before you read it

  • 30 seconds: the live interactive report — https://tjohnson012.github.io/whetstone/ — scrub the ladder, replay the token race, open the three-silicon matrix.
  • 2 minutes: watch the reusable CI gate hold a planted regression permanently red: https://github.com/tjohnson012/whetstone/pull/1
  • 5 minutes, no Arm hardware needed: dispatch the bench workflow in the repo — GitHub's own arm64 runners rebuild everything and regenerate our numbers on machines we do not control.
  • 15 minutes on any Apple Silicon Mac: docs/SETUP-ARM64.md from git clone to reproducing the headline finding live (cold-start verified: every command, first try).
  • docs/JUDGES.md is this menu, in-repo.

What it is

Whetstone is an optimization instrument for LLM inference on Arm CPUs. One model (Qwen3-4B-Instruct-2507), one pinned llama.cpp commit, a seven-rung ladder of Arm-specific optimizations — generic build → aarch64 runtime repacking → KleidiAI kernels → quantization sweep → quantized KV cache → speculative decoding → thread tuning — each rung measured the same way: raw samples, P50/P95, TTFT, per-token traces, peak RSS, and perplexity, with the verbatim reproduction command stored next to every number.

Mobile AI, on the mobile device developers actually carry: an Arm-powered laptop, fully on-device, offline, nothing leaves the machine. Track fit is confirmed in writing — the track targets "Arm-powered client devices such as smartphones, tablets, and laptops," and the organizer answered the exact Apple Silicon question in the challenge forum: "Yes, optimizations for Apple Silicon does count!" (https://arm-ai-optimization-challenge.devpost.com/forum_topics/44631-does-an-apple-silicon-mac-qualify-for-the-mobile-ai-track)

The results (Apple M1 Max, p50 of 5 reps, full protocol)

  • Generation 27.9 → 47.7 tok/s (1.71x) at identical output quality (greedy speculative verification is exact; 88% draft acceptance measured).
  • Prefill 134 → 196 tok/s at the best thread configuration; TTFT 2.37 s → 1.89 s from the first rung alone.
  • Every quant carries its quality bill: Q4_0 is +1.82% perplexity vs an F16 reference measured on identical wikitext-2 chunks. Speed claims without quality evidence are how benchmarks lie; ours can't.

The finding (this is the part worth your three minutes)

Enabling KleidiAI — Arm's recommended acceleration library — triples token-generation latency on the M1 Max (10.5 vs 31.9 tok/s against llama.cpp's own repack kernels; same commit, same model, same flags), while the same binary wins on Neoverse N2 (60.8 vs 55.3 gen, 232.5 vs 153.0 prefill).

So we formed the obvious rule — dotprod-only silicon prefers repack, i8mm silicon prefers KleidiAI — and then tested it on a third chip. It falsified our rule. AWS Graviton2 (Neoverse N1) is dotprod-only exactly like the M1, and KleidiAI wins both axes there (228.6 vs 135.9 prefill, 49.4 vs 45.1 gen; TTFT 3,733 → 451 ms vs the generic build). The collapse is a property of M1-class cores, not of CPU feature flags. Arm Performix code_hotspots profiles on the Graviton instance provide kernel-level receipts: each build's self-time sits in exactly the kernels it claims (kai_run_matmul_*_neon_dotprod vs ggml_gemm/gemv_q8_0_4x4_q8_0).

The fast path is a property of the silicon — not the library, not its feature flags — and nothing warns you. That is why the deliverable is an instrument, not a number: Whetstone measures the ladder on your machine, auto-selects the host-best kernel path, keeps both isolation rungs visible as ground truth, and documents its own heuristic's failure honestly. The regression is reported upstream: https://github.com/ggml-org/llama.cpp/issues/27078.

The N2 ranking was re-validated on 4-vCPU public runners on submission day — same result, wider margin (461.9/101.8 vs 304.5/89.7). Cloud-host numbers are 0.6B quick-protocol runs — the kernel ranking is the claim there, never absolute cross-host numbers; the UI badges every row with its model and protocol.

What ships (all MIT, all reusable)

  1. The Instrument — hand-built web app, no chart libraries, no network calls: ladder scrubber that morphs every number, token race replaying real per-token timestamps, sample-level distributions, quant frontier, three-silicon kernel matrix. Exports to a single self-contained report.html.
  2. whetstone CLI — stdlib-only Python benchmark harness, macOS/Linux arm64, schema-validated session JSON.
  3. whetstone-gate — composite GitHub Action failing PRs on p50 regressions (uses: tjohnson012/whetstone/ci/gate-action@v1); PR #1 is its permanent live demonstration.
  4. Session schema + report exporter + table renderer — the README's own result tables are generated from the measured JSON by script; prose cannot drift from data.

How we measured

Three pinned builds of llama.cpp 74ce157 (generic / repack-only / KleidiAI — the third exists because KleidiAI has no runtime off-switch at this commit, which we document, along with two more pitfalls: quantized V-cache requires flash-attn on CPU, and -md without --spec-type silently never drafts). pp512/tg128, 5 reps after warmup; TTFT and traces from llama-server's SSE stream; peak RSS from /usr/bin/time; perplexity with matched-chunk F16 reference. Full protocol, threats to validity, and explicit non-claims: docs/METHODOLOGY.md.

Built With

Share this project:

Updates