ArmTune

Inspiration Arm cloud and llama.cpp make efficient inference possible, but picking the right threads, batch sizes, and KV-cache settings is still guesswork. Arm Performix and KleidiAI help profile and accelerate, yet developers still need a closed loop: measure a real workload, try configs, and keep the winner. We built ArmTune to turn that into a one-command workflow with hard before/after numbers.

What it does ArmTune is a workload-aware autotuner for llama.cpp on Arm64. It launches candidates with different thread counts, batch/ubatch sizes, and K/V cache types, benchmarks the same prompts, and ranks them by throughput, TTFT, and latency. It emits results.json, a Markdown report with winning flags, and an optional Control Room dashboard to visualize uplift. On Apple Silicon Arm64 CPU we measured 106.95 → 158.18 tok/s (+47.9%) and faster TTFT/latency for Qwen2.5-0.5B.

How we built it Python CLI (armtune) manages llama-server lifecycle, streaming chat benchmarks, scoring, and reporting. Search spaces live in TOML (examples/mac-arm64.toml, quickstart.toml). We built llama.cpp for Arm (Metal on Mac; KleidiAI-ready on Linux Arm64), ran reproducible CPU-only searches (--n-gpu-layers 0), and shipped a React Control Room that loads real results.json. Artifacts: https://github.com/RiyaTallsss/arm-inference-autotuner — Apache-2.0.

Challenges we ran into Oracle Free Tier signup charged a verification hold but never delivered an activation email, blocking Ampere access. An AWS Graviton VM was started, but SSH later timed out (instance/IP/security-group drift). We pivoted to real Arm64 evidence on Apple Silicon, documented the hardware clearly, and kept the cloud path ready for when access returns.

Accomplishments that we're proud of A working end-to-end autotuner with auditable raw samples, a clear +47.9% Arm64 win, submission-ready reports, and a judge-friendly dashboard—not just a blog post of static flags. The first candidate is always the baseline, so improvements are comparable and reproducible.

What we learned On Arm CPUs, “more threads” is not always better; mixed KV-cache types (e.g. q8_0/f16) and moderate parallelism can beat naive defaults. Fair benchmarking needs warmups, fixed prompts, deterministic decoding, and labeled token-count sources. Cloud signup friction is a real part of Arm developer experience.

What's next for ArmTune Confirm the same search on Ampere/Graviton with KleidiAI + Arm Performix profiles, add quality guardrails for aggressive quantization, concurrent-load mode, cost-per-million-tokens estimates, and one-click OCI/AWS provisioning templates so any developer can reproduce an Arm cloud win in minutes.

Built With

• apache2.0
• applesilicon
• arm64
• armperformix
• autotuning
• awsgraviton
• benchmarking
• cloudai
• cmake
• gguf
• github
• kleidiai
• kvcache
• llama.cpp
• llminference
• openaiapi
• oracleampere
• python
• quantization
• react
• recharts
• typescript
• ubuntu
• vite