ArmInfer

• 

Inspiration

AI inference performance can vary dramatically depending on the hardware and runtime configuration being used. When deploying an LLM on Arm64, developers may need to experiment with thread counts, batch sizes, and other inference settings just to determine what performs best on a particular machine.

That led to the idea behind ArmInfer: instead of guessing which configuration should be fastest, why not benchmark the actual Arm64 hardware and let the measurements decide?

We wanted to build a lightweight developer tool that could take an existing llama.cpp inference workload, automatically test different configurations, and produce clear, reproducible evidence showing whether an optimization actually improved performance.

The Arm AI Optimization Challenge gave us the opportunity to explore that idea across two very different Arm64 environments: an Apple M5 development machine and Arm-based infrastructure in Google Cloud.

What it does

ArmInfer is an automated benchmarking and configuration-tuning CLI for llama.cpp inference on Arm64.

A developer provides a GGUF model and ArmInfer handles the benchmarking workflow:

arminfer optimize /path/to/model.gguf

ArmInfer:

1. Detects and validates the Arm64 environment.
2. Identifies available CPU resources.
3. Runs a baseline using llama-bench.
4. Tests multiple thread-count and batch-size configurations.
5. Parses the benchmark results automatically.
6. Compares generation and prompt throughput.
7. Selects the fastest measured configuration.
8. Reports the improvement over baseline.
9. Saves reproducible benchmark evidence as JSON.

ArmInfer also avoids assuming that tuning must always produce an improvement. If none of the tested configurations beats the baseline, the tool can recommend keeping the default configuration instead.

We tested ArmInfer using Qwen2.5-1.5B-Instruct Q4_K_M in CPU-only mode.

Our final Apple M5 benchmark produced:

Metric	Baseline	ArmInfer
Generation throughput	76.27 tok/s	105.28 tok/s
Improvement	—	+38.04%

The fastest measured configuration used 8 threads and a batch size of 512.

We then moved the same project to a Google Cloud Arm64 VM running a Neoverse-N1 processor with 4 vCPUs.

That environment produced:

Metric	Baseline	ArmInfer
Generation throughput	25.85 tok/s	26.12 tok/s
Improvement	—	+1.07%

The best configuration there used 4 threads and a batch size of 128.

The difference between these results became one of the most interesting findings of the project: an optimization that works extremely well on one Arm64 platform may provide only a small improvement on another.

ArmInfer is designed to measure that difference rather than assume one configuration is universally optimal.

How we built it

ArmInfer is a Python CLI that uses llama.cpp and llama-bench as the underlying inference and benchmarking engine.

The project is divided into several small components:

GGUF model
    |
    v
ArmInfer CLI
    |
    +--> Hardware detection
    |
    +--> llama-bench runner
    |
    +--> Benchmark parser
    |
    +--> Configuration tuner
    |
    +--> Results/report generator
    |
    v
Recommended configuration
+ benchmark.json

The hardware layer validates that optimization runs occur on Arm64 and identifies characteristics such as the CPU and available logical cores.

The benchmarking layer invokes llama-bench and consumes its JSON output instead of attempting to estimate performance ourselves.

The tuner establishes a baseline and then evaluates a search matrix containing different CPU thread and batch-size configurations.

The reporting layer turns those measurements into a readable terminal result and a machine-readable JSON artifact.

We also created helper scripts for building llama.cpp, retrieving the recommended GGUF test model, and running a more stable final benchmark with multiple repetitions.

Development began on an Apple M5 Arm64 MacBook. After the local workflow was functional, we cloned the same repository onto a Google Cloud Arm64 Linux instance and repeated the installation and benchmarking process there.

That second environment was especially important because it forced us to validate that ArmInfer was not simply an Apple-specific optimization script.

Challenges we ran into

One of the biggest challenges was discovering how differently the same inference workload behaved across Arm64 environments.

On the Apple M5, configuration tuning produced a substantial 38.04% generation-throughput improvement.

Our first Google Cloud experiment was very different.

On a 2-vCPU Neoverse-N1 instance, the best tested configuration was actually slightly slower than the baseline:

Baseline:     14.62 tok/s
Best tuned:   14.60 tok/s

That result exposed an important flaw in our initial tuner design: ArmInfer assumed that one of the tuned configurations should always become the recommendation.

We changed the design so that the baseline itself can win. If no tested configuration improves performance, ArmInfer can recommend keeping the defaults rather than presenting a slower configuration as an optimization.

Cloud setup also created several practical challenges.

Our first attempt to obtain Arm cloud infrastructure ran into unavailable Ampere capacity. After moving to Google Cloud, we encountered Linux-specific differences that had not appeared during development on macOS.

For example, our initial llama.cpp Linux build lacked HTTPS support, preventing the model helper from retrieving the GGUF model. We traced that to the OpenSSL development dependency, rebuilt llama.cpp, and successfully retrieved the model.

Hardware identification also behaved differently across operating systems. Apple exposed the M5 information directly, while the Linux cloud VM required us to retrieve the processor model through lscpu. That work improved ArmInfer's cross-platform Arm64 hardware detection.

These problems ultimately made the project stronger because they forced us to test the complete workflow outside our original development environment.

Accomplishments that we're proud of

The accomplishment we are most proud of is turning a simple benchmarking idea into a working Arm64 optimization workflow that produced measurable results on two different Arm platforms.

ArmInfer achieved a 38.04% improvement in generation throughput on Apple M5, increasing performance from 76.27 to 105.28 tokens per second in our final benchmark.

We also successfully moved the project from macOS to a real Google Cloud Neoverse-N1 Arm64 environment and ran the same optimization workflow there.

That cross-platform experiment produced a smaller 1.07% improvement, but it validated something equally important: ArmInfer's value is not based on claiming that every machine can receive a huge speedup.

Its job is to determine what actually works on the hardware being tested.

We're also proud that the project produces reproducible benchmark artifacts instead of only printing performance numbers to the terminal. Results can be stored in benchmark.json, making it easier to inspect and compare optimization experiments.

Finally, the project has automated tests covering its core behavior, with 9 tests passing at the end of development.

What we learned

The biggest lesson was that Arm64 is an architecture, not a single performance profile.

Our Apple M5 and Google Cloud Neoverse-N1 machines were both Arm64 systems running the same model through the same underlying inference engine, yet their responses to configuration tuning were dramatically different.

That reinforced the central idea behind ArmInfer: performance optimization should be measured on the target hardware.

We also learned much more about how modern local LLM inference works.

Before this project, concepts such as GGUF quantization, llama.cpp, prompt throughput, generation throughput, batch size, and CPU thread tuning were relatively unfamiliar to us.

Building ArmInfer required understanding how those pieces interact and, more importantly, how to measure them correctly.

Moving from Apple Silicon to Linux in Google Cloud also demonstrated why reproducibility matters. A workflow that succeeds on one operating system can expose missing dependencies, hardware-detection differences, or build assumptions when moved to another environment.

The project changed our perspective from simply asking:

"How fast is this model?"

to asking:

"What configuration makes this model perform best on this specific Arm64 machine, and can we prove it with repeatable measurements?"

What's next for ArmInfer

The current version intentionally focuses on a small, understandable optimization space: CPU thread counts and batch sizes for GGUF models running through llama.cpp.

The next step would be expanding ArmInfer into a broader Arm64 inference optimization toolkit.

Future versions could explore additional llama.cpp parameters, automatically determine more intelligent search ranges based on detected hardware, compare different quantization levels, and evaluate memory usage alongside throughput.

Another important improvement would be adding richer benchmark comparisons across multiple Arm cloud platforms and processors.

We would also like to add historical benchmark storage so developers could compare machines and configurations over time:

Model
  ↓
ArmInfer
  ↓
Hardware detection
  ↓
Automated tuning
  ↓
Benchmark history
  ↓
Platform comparison

Longer term, ArmInfer could evolve from a CLI tuner into a developer tool that answers a simple question:

Given this model and this Arm-powered machine, what is the fastest configuration I should actually use?

The first version demonstrates that the answer can vary substantially across Arm64 hardware—and that automatically measuring it is more reliable than guessing.

Built With

• cmake
• github
• google-cloud
• hugging-face
• linux
• llama.cpp
• python
• qwen