GraviTune

Measure the Arm machine you're actually on. GraviTune found that llama.cpp's default setting costs 2x prefill throughput on AWS Graviton4 proven with hardware counters and filed upstream.

Comment

Inspiration

I had no Arm hardware. That turned out to be the interesting constraint.

The Arm challenge rewards measured optimization. I decided that whatever I built, every number in it would come from real Arm silicon or it wouldn't ship.

Once I had a Graviton4 box running, I asked a question I assumed had a boring answer: are llama.cpp's defaults actually right on Arm? Everyone benchmarks models. Almost nobody benchmarks the settings they inherit them from an x86 laptop and deploy to Graviton.

The answer was not boring.

What it does

GraviTune measures LLM inference on the Arm machine you are actually on, picks the best config for a stated objective, and emits a reusable tuned-config artifact with the evidence attached.

gravitune detect identifies the CPU from MIDR_EL1, the architectural ID register not the cloud instance label. c7g tells you what you were billed for; MIDR tells you which microarchitecture you got, which is what decides whether the optimized kernels can even run.

gravitune tune sweeps threads, micro-batch, flash-attention, and quantization, then writes a tuned config, a ranked report, and every raw measurement.

It found two things that cost real users real time.

1. llama.cpp's default flash-attention setting halves prefill throughput on Graviton4. -fa auto resolves to on. Turning it off:

config prefill tok/s TTFT (512-token prompt)
-fa auto (the default) 430.4 1190 ms
-fa off 854.2 599 ms

1.99x prefill. 1.99x faster time-to-first-token. From one flag.

2. Oversubscribing threads costs 4.5–6.1x on token generation. 32 threads on 16 physical cores drops decode from 127.5 to 28.1 tok/s, while prefill barely moves. Arm server cores have no SMT, so the x86 habit of oversubscribing is a cliff.

It also ships an MCP server, so a coding agent can interrogate the machine it's deployed on instead of guessing from a model card.

How we built it

Stdlib-only Python, deliberately. On a fresh Arm box every pip install is a chance for a missing aarch64 wheel to become a slow source build a tool that promises "one command on a fresh Graviton box" can't need a dependency tree to start.

The sweep engine drives llama-bench, parses its JSON, and records prefill, decode, and derived TTFT per config. Selection is objective-driven, because prefill and decode pull in opposite directions and there is no single "fastest".

I ran three Graviton generations in parallel - c6g (Neoverse-N1), c7g (V1), c8g (V2) each cloning the tool in EC2 user-data and sweeping unattended. Then perf on Graviton4 to find out why, and Arm Performix for target characterization.

CI runs the whole thing on GitHub's free ubuntu-24.04-arm runners (image managed by Arm), so anyone can verify it works on real Arm silicon without provisioning hardware.

Challenges we ran into

The PMU counter limit. Arm Performix's microarchitecture recipes need 3+ general-purpose PMU counters. Graviton VMs expose only 2 the kernel reports 3, but one is the dedicated cycle counter. So code_hotspots and system_utilization work while cpu_microarchitecture and instruction_mix fail. I couldn't find this documented anywhere, so I wrote it up. (Separately: Ubuntu's perf_event_paranoid = 4 blocks unprivileged perf that's a distro default, not an AWS restriction, and it's easy to misread as "cloud VMs don't expose counters.")

My own scoring function was a coin toss. The interactive objective blended decode and TTFT with a weight of 0.15 that I'd picked by feel. Sweeping that weight over the real data showed the winning config flips at w = 0.146853 0.003 away from my constant. Since decode varies ~15% run-to-run, the "recommendation" was noise, and nothing in the output would have revealed it. I replaced it with a stated TTFT budget and shipped scripts/objective_sensitivity.py so anyone can run the same check on their own objective.

A hypothesized prediction in my own docs was wrong. I'd assumed that the flash-attention advantage should reverse at long context, since the attention working set grows quadratically. I measured it: 512 → 8192 tokens, and the advantage grows monotonically from 1.97x to 5.20x. No crossover. I corrected the docs rather than quietly dropping the claim.

Deciding what NOT to claim. Two runs of the same baseline gave decode of 127.5 and 146.8 tok/s a 15% swing. So I dropped the decode claim entirely and kept only prefill/TTFT, which reproduces to 0.05%. Similarly, I refuse to claim "Arm is 9x faster than x86" from 852 vs 96 tok/s different core counts, class, and power envelope make that number meaningless.

Accomplishments that we're proud of

Everything is measured on real silicon, four times over. The 1.97x result reproduced across four independent runs, including a clean-box run of the documented curl | bash command.

I explained the mechanism, not just the number. perf shows flash attention retiring 1.97x more instructions for identical work (389.7B vs 197.9B) matching the throughput gap almost exactly. And a trap worth knowing: the slow path has the higher IPC (4.08 vs 3.49). It efficiently executes work that never needed to happen. IPC measures how well you execute instructions, never whether they should exist.

The finding is a rule, not an anecdote. Across generations the answer inverts: Neoverse-V2 gains 1.99x, V1 gains 1.55x, and N1 gains nothing its smaller caches make the default correct. It tracks the core family, not the generation number. It tracks the core family, not the generation number. On x86 the same flag is a 4.7% loss. If the optimal config were the same everywhere, you wouldn't need an autotuner; you'd need a blog post.

I audited my own tool and published the failure. Finding that my scoring function was balanced 0.003 from a flip point, then shipping the script that proves it, is the thing I'm most pleased with.

What we learned

  • IPC can point the wrong way. Optimizing on IPC alone would have ranked the slow path as the efficient one.
  • A default is only correct in the context it was chosen in. FlashAttention trades arithmetic for memory traffic right on a GPU, wrong on a Neoverse core whose caches already hold the working set.
  • A threshold inside your logic is only meaningful if it separates. Mine didn't, and I only found out because I tested it.
  • Two runs beat one. The second run is what exposed the 15% decode variance and stopped me shipping a claim I couldn't defend.
  • Cloud instance labels are not microarchitecture. MIDR is.

What's next for GraviTune

  • Finish the 7B sweep. It was launched but didn't complete before the deadline, so no 7B claim is made here. Larger models are more bandwidth-bound and should shift the optimum.
  • Find where flash attention finally wins. It hasn't reversed by 8K; a much longer context or far less cache per core should eventually get there.
  • More runtimes ExecuTorch, ONNX Runtime, LiteRT. Only llama.cpp is covered today.
  • Tokens per watt. Arm's real pitch is efficiency, and Graviton guests don't expose energy counters, so tokens-per-dollar is currently the closest proxy.
  • Land the fix upstream properly. The patch is written and verified on Graviton4 and lives in patches/. Doing it right means a regression test and a full local CI run first.

Built With

Share this project:

Updates