Armboost

Self-calibrating llama.cpp serving for Arm CPUs. Probes memory bandwidth, sizes the deployment from the GGUF header, auto-finds the latency knee under open-loop load.

Comment

Inspiration

This didn't start with AI.

I've been porting Mercenaries: Playground of Destruction (Xbox, 2005) to the PS Vita. The Vita runs an ARM Cortex-A9, so I spent weeks on ARM compilation. -mcpu targets, PGO, LTO, BOLT.

That got me wondering how people run language models on ARM. Turns out it's mostly guesswork. llama.cpp gives you about twenty flags that change how it serves and no guidance on any of them. Every blog post has different numbers and none explain why theirs are right, or whether they'd hold on a different sized machine.

The aim, put bluntly, was to optimise AI with AI with zero experience. I know vibecoding is a dirty word and it winds up real engineers. So I treated it as a constraint. I worked up from basic flags instead of reading what frontier LLM research already says, partly because most of it would have gone over my head, mostly because I wanted to see what I'd find without knowing what I was meant to find.

What it does

You give it a GGUF and a machine. It measures that machine's memory bandwidth, reads the model's shape out of the GGUF header, works out how many llama-server processes to run and how wide to pin each one, starts them behind nginx, then finds the request rate where latency is still acceptable.

Nothing is hand-configured. Core count comes from the host, model shape from the header, bandwidth from a probe. There's no config file and no machine profile to pick, so pointing it at a different sized box gives you a different layout.

It also refuses to report numbers it can't stand behind. Model hash, CPU features, huge page state, source commit and the launched layout versus the planned one all get checked, client and server token counts are reconciled separately, and any failed request invalidates the cell.

How we built it

Started with simple compiler flags, one at a time. Most did nothing. PGO, BOLT, -ftracer, all null or negative once measured properly. Demoralising until I worked out the null results were the useful part.

What worked was going up a level. Measure the machine's bandwidth, what one process gets alone versus what they all get together. Read bytes-per-forward from the GGUF header. Derive the layout from those two. Put the operating point below the cliff with square root staffing, a queueing result from 1981. Prove it with a load ladder.

All measured against a real llama-server under open loop load at constant arrival rate, so it keeps offering requests whether the server is coping or not. A closed loop client can't show you a queue. It waits for a reply before sending the next request, so it never overloads anything.

Challenges we ran into

Believing my own results, and it stayed the hard part.

Early on I found ub=48 was the optimal micro-batch. I'd predicted it beforehand and my own script printed "PREDICTION CONFIRMED". Wrong. The five configs spanned 1.2% while repeat runs of the same config varied 2.5%. The differences were inside the noise. max() always returns something, it always looks like a result, and it's most convincing when it agrees with what you already thought. The real finding was a plateau.

That happened three seperate times. An argmax over noise, a "decode speedup" that was an artefact of how wall time gets attributed, and a throughput number that was an artefact of my measurement window.

Then finding where it stops working. The constants were fitted on Google Axion. They transferred to Neoverse-N3 and missed by 2x on Neoverse-N1, which hasn't got i8mm.

I also tried pushing the concurrency work onto mobile. Ran it on a Snapdragon, a Galaxy S24, splitting across CPU, GPU and NPU, and got just over 40 tokens a second. It worked, and the partitioning ideas carried across. But the sizing constants for it was never properly calibrated. The thermal derate is a guess because phones throttle and I never measured by how much. So I fixed on the CPU scaling harness and left mobile out rather than ship numbers I couldn't stand behind.

Accomplishments that we're proud of

Two findings I haven't seen written down anywhere.

There's a discontinuity at llama.cpp's 64-row flash attention tile threshold. Moving n_ubatch from 63 to 64 puts you on the tiled path and costs about 37% throughput on Graviton3. One row. Not a curve with a gentle optimum, a cliff at a hardcoded constant. It means ubatch is a kernel-path selector, not a dial. That's Graviton3 evidence though, it didn't carry to the final Axion config, and the shipped build uses the stock threshold and asserts it rather than patching it.

The second is that you can compute the operating point instead of sweeping for it. Bandwidth and bytes-per-forward give service time, square root staffing gives a safe request rate, and across 16 to 72 cores it lands within about 2% of the measured knee. It works because a batching inference server behaves like a bulk service queue, which was solved in 1954.

The other thing I'm pleased with is that each of those retractions became a check in the code rather than a note in a file. It refuses to name a winner when the gap is smaller than the noise, throws away the first cell after startup, and reports its own measurement resolution so I can't quote a difference finer than it can resolve. And it reports calibration-required on hardware it hasn't seen, because a sizing tool that confidently gives the wrong answer on unfamiliar silicon is worse than none.

What we learned

Throughput is the wrong target. A server can report healthy tokens per second while users wait seconds for their first word. Throughput tails off gently, latency falls off a cliff. On a 32 core box, 11 to 13 requests a second takes p95 time-to-first-token from 65ms to 2.4 seconds and throughput barely moves.

Nine times the cores gives 5.6x the throughput, not nine. Per-core output drops from 56 to 35 tokens a second as width grows because the processes share one memory system.

Working up from nothing means rediscovering plenty that's already established. Everyone knows bandwidth is the constraint on CPU inference and I got there the long way round. But you don't inherit anyone's assumptions either, and a few things I measured didn't match what gets repeated online.

I still couldn't write this code unaided. The coding was never the hard part though. Getting a straight answer out of the measurements was.

What's next for Armboost

GPU and CPU concurrency on the same box. I've done some of this already on a g5g instance, partitioning cores between a CPU tier and a GPU tier and measuring whether they interfere. Static partitioning with the right polling behaviour held both tiers at their knees with no measurable interference, which surprised me, and a work-stealing approach I expected to be better turned out worse. The interesting case is a machine where the GPU is saturated and 70 idle CPU cores are sat next to it doing nothing.

I had GH200 access lined up to take that further and ran out of time. Nothing from it made it into this submission, which is why there are no GH200 numbers anywhere in the repo. That's the first thing I'd pick up.

Then mobile properly. The Snapdragon work split across CPU, GPU and NPU and produced a real number, but the constants behind it were never calibrated so it sits outside the results. Measured thermal behaviour instead of a guessed derate would let the same harness size a phone the way it sizes a server, across all three compute units rather than treating them separately.

There's unfinished business in the CPU results too. The 48-core row never got a failing rung above it, so its knee is a lower bound rather than a measurement, and the 8-core box sits outside the fit in a way I don't understand yet. And while 7B runs fine under serving load and the 235B answers single-stream, getting that 235B under concurrent serving load hasn't worked yet. That's the piece I most want to finish.

Built With

Share this project:

Updates

Submission history