Track: Cloud AI
Inspiration
We benchmarked llama.cpp on a GCP Axion (Neoverse-V2) instance and found something counterintuitive: the slowest GGUF quantization format on a generic armv8-a build (Q4_0, 82.2 tok/s prompt processing) becomes the fastest once the build enables i8mm/SVE2 (310.5 tok/s) — a 3.78x gap from build flags alone, and a complete flip of the format ranking. Configuration wisdom imported from x86 is not just suboptimal on Arm — it can be exactly backwards. That measurement is the whole reason this tool exists.
What it does
armtune is a zero-dependency Python CLI that (1) probes the CPU for Arm features (i8mm, SVE, SVE2, asimddp, bf16), (2) runs a llama-bench matrix across your models x thread counts, and (3) emits a Markdown report with the full results matrix, links to raw benchmark outputs, the best configuration, best-vs-worst speedup, and a copy-paste reproduce command. One command turns "which quant should I deploy?" into a measured answer.
How we built it
Pure Python 3.10+ standard library (subprocess + json + argparse), so it runs on a fresh cloud instance with no pip install. Validated end-to-end on a GCP Axion c4a-standard-8: the unedited report and raw outputs are committed in examples/axion-c4a-8/, and the 18-run baseline-vs-native study that motivated the tool is in examples/axion-c4a-8/study/ with the exact script (scripts/axion_bench.sh) that produced it. Unit tests use the real /proc/cpuinfo captured from that machine and pass on non-Arm machines too (probing degrades gracefully).
Challenges
llama-bench emits separate JSON rows for prompt and generation throughput, single combos can fail (OOM, bad model) without invalidating the matrix, and non-Arm dev machines must still run the test suite. The runner records per-combo failures and continues; the prober returns explicit unknowns instead of guessing.
What we learned
On Neoverse-V2, runtime repacking of Q4_0 into aarch64-optimized layouts (feeding i8mm matrix multiply) beats both Q4_K_M and Q8_0 at prompt processing — and thread scaling from 4 to 8 cores nearly doubles throughput. None of that is visible from model cards; you have to measure the machine.
What's next
Auto-detecting NUMA/core topology for smarter thread sweeps, KV-cache quantization in the matrix, and a cost-per-million-tokens column using cloud pricing.
Log in or sign up for Devpost to join the conversation.