armtune

Track: Cloud AI

Inspiration

Running an LLM on an Arm server can be several times faster than the configuration most people actually deploy, and the win is basically free. The problem is that the fast path is scattered across a dozen settings: the build has to turn on the Arm dot product and 8 bit matmul extensions and the KleidiAI kernels, the weights have to be quantized to a type those kernels like, and the thread count, flash attention and KV cache precision all interact with the specific machine. Nobody wants to benchmark that grid by hand, so the default is to leave most of the performance on the table. We wanted a tool that finds the fast config for you and, just as importantly, writes down the evidence.

What it does

armtune takes a small model, builds llama.cpp two different ways (a plain portable build and an Arm build with dotprod, i8mm and KleidiAI), quantizes the model to several types, and runs a staged search over build variant, quantization, thread count, flash attention and KV cache precision. It returns the fastest configuration for your chosen objective and emits three artifacts: a human readable before and after report, a machine readable results file, and a small deploy ready lockfile with the exact command to serve the winning config. If matplotlib is present it also draws two charts.

On a GitHub hosted ubuntu-24.04-arm runner (4 vCPU, Neoverse V2 class core), a full tune of Qwen2.5 0.5B Instruct produced:

• decode speed up 5.03x, from 23.1 to 116.2 tokens per second
• model size down 66 percent, from 1,202 MB to 403 MB
• holding the quantization fixed, the Arm build alone beat the portable build by 2.18x on decode and close to 6x on prefill

How we built it

The core is a small, dependency free Python package. It detects the host CPU and its Arm features, fetches and builds a pinned llama.cpp in two configurations, downloads a base GGUF model and derives quantized variants, then drives llama-bench in JSON mode to get clean prefill and decode numbers.

The search is coordinate descent across five stages, so a full sweep is under twenty benchmarks instead of the hundreds a full grid would need. Stage one picks the build variant at a fixed quant so the Arm build win is measured on its own. The later stages lock in quantization, thread count, flash attention and KV cache precision in turn. Every trial is kept, which lets the report show the Pareto front of size against speed for people who care more about memory than raw throughput.

The whole benchmark runs in GitHub Actions on free Arm64 runners, so the results are reproducible by anyone with no special hardware. The numbers in the repo were committed straight from that workflow.

Challenges we ran into

Isolating the Arm build win was the interesting part. To show that the build flags matter on their own, we had to benchmark the same quantization on both the portable and the Arm build rather than only comparing the final tuned config to the baseline. Getting stable numbers also meant running llama-bench with warmups and repeats and parsing its JSON rather than scraping text.

Accomplishments that we are proud of

The result is honest and reproducible. Every number comes from a real Arm64 machine and can be regenerated by pressing one button in the Actions tab. The tool is also genuinely reusable: it works on any small GGUF model, not just the default, and the output lockfile is something you can actually deploy.

What we learned

Most of the Arm inference speedup is in the build, not just the quantization. Switching from a portable build to one that enables dotprod, i8mm and KleidiAI was worth more than 2x on decode at the same model size, which is a bigger lever than people expect.

What's next for armtune

More knobs (batch size and NUMA placement), larger models, and an optional quality check so the report can show speed next to a perplexity delta.

How to set up and validate on Arm64 (for judges)

Option A, no hardware needed:

1. Open the repo Actions tab.
2. Select the benchmark-arm64 workflow and press Run workflow.
3. When it finishes, the report is in the run summary and the full results directory is attached as an artifact.

Option B, on your own Arm64 machine:

git clone https://github.com/businessarshgoyal/armtune.git
cd armtune
pip install -e ".[charts]"
armtune info
armtune tune --objective latency

The before and after report lands in results/report.md.

Built With

• arm64
• dotprod
• gguf
• github-actions
• i8mm
• kleidiai
• llama.cpp
• python
• quantization