Arm64 vs x86 LLM Inference Benchmark

Inspiration

More AI infrastructure is moving toward Arm-based chips in the cloud they're cheaper to run and more power-efficient than traditional x86 servers. But most AI tooling and benchmarks are still written and tested with x86 in mind first. I wanted to actually test the assumption everyone makes: does a standard LLM inference stack run well on Arm out of the box? I didn't want to just guess I wanted real, measured numbers.

What it does

This project benchmarks LLM inference performance using llama.cpp and a small open-source model (TinyLlama 1.1B, Q4_K_M quantization) across two cloud CPU architectures "Arm64" and "x86" using GitHub Actions' free hosted runners. It runs controlled benchmarks at fixed thread counts (2 and 4) so the comparison is fair, then tests whether Arm-specific compiler optimizations can improve performance further.

How we built it

I used GitHub Actions as the entire cloud environment no AWS, no paid cloud account, just free hosted runners (ubuntu-24.04-arm for Arm64, ubuntu-24.04 for x86). The workflow:

1. Builds llama.cpp from source
2. Downloads a small quantized LLM (TinyLlama)
3. Runs llama-bench at fixed thread counts to measure prompt-processing speed (pp512) and text-generation speed (tg128)
4. Repeats the same test after applying Arm-specific build flags (-mcpu=native, then GGML_NATIVE=ON) to see if performance improves

Every run is logged directly in GitHub Actions, so the whole benchmarking history is public, reproducible, and free for anyone to re-run themselves.

Challenges we ran into

• My initial "Arm is faster" result turned out to be a false signal caused by uncontrolled thread counts a good reminder to double-check assumptions before trusting a benchmark.
• A dependency I relied on (huggingface-cli) was deprecated mid-project and had to be swapped for its replacement (hf) after a failed run pointed me to the fix.
• Getting genuinely fair, apples-to-apples numbers took several iterations fixing thread counts, confirming runner architecture explicitly in the logs (rather than trusting job names), and re-running tests to rule out noise.

Accomplishments that we're proud of

• Built a fully free, reproducible benchmarking pipeline for comparing Arm64 and x86 cloud performance with zero cloud account signups, zero cost, and zero local hardware required.
• Caught and fixed a real methodology flaw mid-project: an early result made it look like Arm64 was faster, but that was actually due to uncontrolled thread counts between the two runner types. Fixing that and re-running the tests produced an honest, defensible comparison instead of a misleading one.
• Delivered a real, evidence-backed finding rather than an assumption including the fact that manual Arm-specific compiler flags made no measurable difference, which is a useful and honest result in its own right.
• Debugged and resolved a real-world tooling issue (the deprecated huggingface-cli → hf migration) using the actual CI failure logs, without needing local development access.

What we learned

• Real benchmarking requires controlling variables. My first runs looked like Arm was faster than x86 until I realized the two runner types were defaulting to different thread counts. Once I fixed both at the same thread count, the result actually reversed: x86 outperformed Arm64 on this workload.
• Not every optimization attempt works, and that's still a valid result. I tried two different ways of forcing Arm-specific compiler optimizations. Neither changed performance at all. That told me llama.cpp's default build already uses Arm's native CPU features effectively — the performance gap comes from the underlying hardware of the free runners, not unused software potential.
• You don't need a cloud account to do real cloud benchmarking. GitHub's free Arm64 and x86 hosted runners were enough to run a legitimate, controlled architecture comparison from scratch.

What's next for Arm64 vs x86 LLM Inference Benchmark

• Test larger models and additional quantization levels (e.g., Q8_0, F16) to see if the Arm-vs-x86 performance gap changes with model size.
• Benchmark other inference engines beyond llama.cpp (e.g., ONNX Runtime, vLLM) to see if the pattern holds across different tooling.
• Investigate the specific underlying CPU hardware behind GitHub's Arm64 and x86 runners to better explain why the performance gap exists at the hardware level.
• Extend the benchmark matrix to cost-per-token, not just raw speed — since Arm instances are often cheaper, a slower-but-cheaper result could still be the more practical choice for real deployments.
• Package the benchmarking workflow as a reusable GitHub Action so other developers can drop it into their own repos to test their own models on Arm vs x86 with one click.

Built With

• arm64
• cloud-computing
• cmake
• github-actions
• huggingface
• llama.cpp
• llm
• tinyllama
• ubuntu