armtune — the story

Inspiration

Every guide for running LLMs on Android says the same things: use the big cores, enable KleidiAI, match threads to core count. I wanted to know what was actually true on the phone in my hand — measured, not assumed. So I built a tool that races configurations against each other and reports what wins on your device. Along the way, it uncovered something much bigger than a tuning script.

The discovery that became an upstream fix

My early tuning runs made no sense: llama.cpp's core-pinning flags (-C, --cpu-strict) changed nothing. Same numbers with or without them. After ruling out my own setup, I went into llama.cpp's source and found the reason — the affinity code was behind a compiler check that is true on desktop Linux but never true on Android. So on every Android phone, for about two years, those flags were accepted and silently ignored.

The fix is one line:

-#elif defined(__gnu_linux__)
+#elif defined(__linux__)

I filed the issue and submitted the patch during this challenge, as sole contributor. It was merged upstream as llama.cpp PR #26838, approved by @ggerganov (2026-08-10). Every llama.cpp user on Android benefits, whether or not they ever run armtune.

armtune stays honest about this history: instead of trusting the flags, it verifies that pinning is actually applied on your binary, and falls back to taskset on older builds — reporting which path it took.

The KleidiAI surprise

With pinning finally working, a second anomaly showed up: a KleidiAI-enabled build decoded much slower on efficiency cores than a plain build — up to −32% on the little cluster and −64% in the default all-core invocation on one test device. The cause is in how llama.cpp picks between kernel paths (KleidiAI's path gets selected ahead of a faster one for this specific workload shape), not in KleidiAI's kernels themselves. armtune detects whether your build includes KleidiAI and runs a live A/B on your hardware, so you get a measured answer instead of a guess.

How I built it

armtune is one self-contained script that runs over adb — nothing installed on the phone, a full run in under 15 minutes:

1. Probe the CPU cluster layout automatically — no hardcoded masks.
2. Verify that affinity flags really pin threads before trusting them.
3. Sweep each cluster configuration against the untuned default.
4. Validate the winner end-to-end on three real workloads (chat, balanced, summarize).
5. Report copy-paste flags plus an HTML report with every number's session provenance.

I leaned on AI tooling (Claude) as a pair programmer for the script and report generation. That combination is what let a one-person project ship a tuner, a findings write-up, and a merged llama.cpp contribution inside the challenge window.

Results

On a Realme RMX3842 (4×Cortex-A78 + 4×Cortex-A55, Android 16, Qwen2.5-0.5B-Instruct Q4_K_M): +23% decode throughput, validated end-to-end — 1.24× chat, 1.22× balanced, 1.23× summarize. And on a phone where all cores are effectively equal, armtune reports the other honest answer: defaults are already optimal, no flags needed.

Challenges

Benchmarking on a phone is unforgiving: background activity, thermal throttling, and leftover processes can all quietly corrupt a run. Several early results looked great and turned out to be measurement artifacts — including one late run that showed a huge "speedup" only because the baseline had thermally throttled. armtune now ships the guards that caught these: it checks the device is idle before each run, kills stale benchmark processes, and flags a report when the baseline looks compromised instead of publishing a misleading headline. FINDINGS.md keeps the full history of what changed and why.

What I learned

• A flag being accepted doesn't mean it does anything — verify the behavior, not the option.
• A speedup is only as trustworthy as its baseline.
• Sometimes the most valuable output of a tuning tool is "your device doesn't need tuning."

What's next

Test on more devices — different SoC layouts are exactly where an auto-tuner earns its keep. Deepen the KleidiAI coverage: run the A/B across more builds, quantizations, and workload shapes to map where it helps and where it hurts. Add more benchmarking options (more models, longer contexts, battery-focused runs). And keep maintaining armtune as an open-source tool — issues and contributions welcome, and if the KleidiAI behavior holds up across more hardware, report it upstream the same way the affinity bug was.

Built With

• html
• shell