LLM inference optimization on Arm using KleidiAI

A reproducible Arm64 study of llama.cpp + KleidiAI on Android, measuring how Arm CPU optimizations affect prompt processing and token generation across LLM workloads.

Comment

Inspiration

Running an LLM locally on a phone is very different from running one on a desktop or cloud server. CPU capabilities, available instruction sets, memory constraints, and the type of workload can all affect inference performance.

I wanted to investigate a more specific question: when Arm-specific optimizations such as KleidiAI are enabled in llama.cpp, do they actually improve every part of LLM inference?

Instead of focusing on one headline benchmark number, I wanted to measure two different inference workloads, prompt processing and token generation, on a real Android ARM64 device.

That led to a project focused not just on finding speedups, but on understanding where an optimization helps, where it does not, and how confidently those results can be interpreted.

What it does

The project benchmarks llama.cpp with KleidiAI disabled and enabled on an Android ARM64 device.

It measures two workloads:

  • pp512: prompt processing
  • tg128: token generation

The experiments use Q4_0 versions of Qwen models and compare otherwise equivalent llama.cpp builds.

The tested device is ARM64/aarch64 with DotProd support. i8mm, SVE, and SME are unavailable on the device.

The project preserves the raw benchmark output and provides scripts and documentation for reproducing and inspecting the experiment.

The key finding is that KleidiAI does not produce a universal speedup. Its measured effect varies with the model size and inference workload.

For example, on the 0.5B model, KleidiAI was slower in both measured workloads. On the 1.5B model, prompt processing was faster in the captured experiments, while token generation was more variable.

The project therefore treats workload-dependent behavior as the result rather than reducing everything to a single "overall speedup."

How we built it

The project was built around llama.cpp commit 1d2869c.

Two configurations were benchmarked:

  1. A baseline llama.cpp build with KleidiAI disabled.
  2. A KleidiAI-enabled llama.cpp build.

Both configurations were run on the same Android ARM64 environment using the same Q4_0 models.

The benchmark workflow uses llama-bench and records the raw output for each configuration. The repository then organizes those measurements into:

  • raw benchmark logs
  • summarized results
  • build and benchmark scripts
  • an experiment log
  • Arm-specific technical notes
  • reproducibility documentation

The hardware capabilities were also recorded because the available Arm instruction set features are relevant when interpreting the optimization behavior.

The project intentionally keeps measured results separate from hypotheses about why those results occur. When the underlying mechanism has not been directly profiled or experimentally verified, it is presented as a hypothesis rather than a conclusion.

Challenges we ran into

The biggest challenge was realizing that benchmarking an optimization is not as simple as comparing two final numbers.

The Android device has DotProd support but does not expose i8mm, SVE, or SME. This limits what can be inferred about behavior on other Arm processors.

Another challenge was measurement variability. The 1.5B model produced substantially different absolute throughput between two captured benchmark sessions. Rather than averaging those captures together or selecting the most favorable result, they are preserved and clearly distinguished.

This made the final story less straightforward than simply saying "KleidiAI makes inference faster," but it also made the experiment more honest.

A further challenge was separating measured behavior from its possible causes. Without dedicated profiling and broader hardware testing, it would be inappropriate to claim a specific kernel or instruction-level explanation for every observed difference.

Accomplishments that we're proud of

The most important accomplishment is that the project became an auditable optimization study rather than just a benchmark screenshot.

The repository contains the raw benchmark logs alongside the scripts and documentation used to build and evaluate the configurations.

We are particularly proud of preserving negative results. The 0.5B experiment did not show a speedup with KleidiAI, and that result remains part of the project rather than being hidden in favor of the more favorable 1.5B measurements.

We also preserved separate 1.5B captures instead of combining measurements that were not directly comparable.

This makes the project useful as a small case study in how Arm-specific optimization should be evaluated: by workload, model size, and actual hardware rather than assuming an optimization is universally beneficial.

What we learned

The biggest lesson was that LLM inference performance is workload-dependent.

Prompt processing and token generation are different computational workloads, so an optimization can affect them differently.

We also learned that a benchmark result is only as useful as its provenance. Preserving the raw output, documenting the exact build, recording hardware capabilities, and distinguishing measurements from hypotheses makes the result much easier to inspect and reproduce.

Most importantly, we learned that negative or inconsistent results are valuable. An optimization that does not improve a particular workload is still an important observation when it is measured carefully.

This changed our perspective from:

"How much faster did KleidiAI make the model?"

to:

"Under which workload and hardware conditions does KleidiAI change performance, and by how much?"

What's next for LLM inference optimization on Arm using KleidiAI

The current results establish that KleidiAI's effect is workload-dependent - not why. Three next steps follow directly from that gap:

  1. Profile the execution paths. Right now the difference between prompt processing and token generation behavior is inferred from throughput numbers alone, not from what's actually happening inside the kernel calls. Profiling llama.cpp's execution path under both configurations would show which operations KleidiAI is accelerating and which it isn't turning "we observed X" into "we observed X because Y."

  2. Test on hardware with i8mm or SVE support. The current device only exposes DotProd, so these results describe one point on the Arm feature spectrum, not Arm CPUs broadly. Running the same benchmark suite on a device with i8mm or SVE would show whether the workload-dependence is a DotProd-specific artifact or a more general KleidiAI behavior - this is the single biggest limitation of the current study, so it's the highest-priority extension.

  3. Increase benchmark captures and control for thermal/background load. The 1.5B model already showed meaningfully different throughput across two captured sessions, which means the current numbers include unmeasured noise from thermal throttling or background processes. More independent runs under controlled conditions would tighten the confidence in every other result, including the profiling and cross-device work above.

The current project deliberately stops before making those claims. The measurements establish that the effect is workload-dependent on the tested device. The next stage is to determine why.

Built With

Share this project:

Updates

Submission history