Inspiration
This project started with a hypothesis that turned out to be wrong, and then two more.
I set out to find something slow about running AI agents on Arm servers. My first idea was that llama.cpp wasn't reaching Arm's i8mm int8 matrix kernels during token generation. I read the KleidiAI dispatch code and found it ships an mr=1 dotprod kernel specifically for batch-1 decode. Dispatch was correct. I had described intended behaviour as a defect.
My second idea was that agent servers re-prefill their giant system prompt on every turn. That one was at least testable, so before building anything I wrote a 60-line probe. It came back in 45 minutes: TTFT dropped 811 ms → 327 ms across turns. Prefix caching already worked. That probe cancelled about two weeks of planned work on a problem that did not exist.
So I had no project and a deadline. But the probe left a residue. Caching worked for one user. Nobody benchmarks two.
I opened a second tenant against the same server, and time-to-first-token went from 386 ms to 4204 ms.
What it does
It identifies, measures, explains, and fixes a 10.9x time-to-first-token regression that appears in llama.cpp the moment a second tenant shares one agent — and it tells you whether your workload is even affected.
An AI agent's prompt has a distinctive shape:
[ system prompt + tool schemas ][ conversation history ][ new turn ]
large, and byte-identical per-tenant
across every user of the agent
llama-server keeps conversations in slots and caches each slot's KV state. To route an incoming request it scores every slot by longest-common-prefix similarity and accepts any slot that clears --slot-prompt-similarity — which defaults to 0.10.
Similarity is roughly \( \frac{\text{shared prefix}}{\text{total prompt}} \). Because every tenant carries the same 550-token preamble, any two tenants of the same agent are already 0.6–0.9 similar to each other. Against a threshold of 0.10, every slot looks like a valid match for every tenant. Requests scatter onto foreign slots, each one recomputes its history from scratch, and each eviction destroys the cache of whoever was there. It is a self-sustaining thrash — it does not warm up and decay, it stays.
Measured on Microsoft Cobalt 100 (Arm Neoverse N2) via Azure, 4 vCPU, CPU-only:
| 1 tenant | 4 tenants, default | 4 tenants, fixed | |
|---|---|---|---|
| Time to first token | 386 ms | 4204 ms (10.9x) | 399 ms |
| Tokens recomputed / request | 35 | 220 | 36 |
| Slot chosen at similarity | 0.955 | 0.716 | 0.955 |
The fix is one flag — --slot-prompt-similarity 0.9. No patch, no rebuild, no code change, and output is byte-identical (sha256 6cdd292a39a33d60… across four thresholds).
But the flag does not matter for every workload, and that is the more useful half of the result. I swept six prompt shapes across the full threshold range:
| prompt shape | measured inter-tenant similarity | recomputation swing |
|---|---|---|
| RAG, large per-tenant document | 0.121 | 1.00x — flat |
| prefix20 | 0.169 | 1.00x — flat |
| prefix35 | 0.288 | 1.00x — flat |
| prefix50 | 0.414 | 1.00x — flat |
| prefix65 | 0.543 | 16.94x |
| Agent, shared system prompt + tools | 0.716 | 6.29x |
Below roughly 0.5 inter-tenant similarity the routing threshold changes which slot is picked and changes no work at all. Above it, the wrong value costs 6–17x. The four prefix* shapes hold total prompt length constant and vary only the shared fraction, so this is not "shorter prompts are faster."
And the cliff sits at each shape's own similarity: 0.543 drops between thresholds 0.5 and 0.7; 0.716 drops between 0.7 and 0.8. That relationship was written into tools/workloads.py as a prediction before the run, and it held on a prompt shape that did not exist when the prediction was written.
So this ships as a screening rule, not a magic number. tools/parse_slot_log.py reads llama-server's own log and tells you the inter-tenant similarity your workload actually has, before you tune anything. Under ~0.5, don't bother. Over it, set the threshold above your measured similarity.
How we built it
Zero third-party dependencies — Python 3 standard library only, so a judge needs no environment to reproduce it.
The adjudicator came first. tools/skeptic.py returns VERIFIED / UNCERTAIN / REJECTED against fixed criteria: ≥5 repeats, relative standard deviation ≤10%, effect ≥5%, and disjoint 95% confidence intervals. It was committed in 20a6031 before the first measurement existed, and it did not change while the hypothesis changed three times. It has unit tests against known answers, because an adjudicator that is only run on real data is never actually checked.
The harness starts a fresh server for every repeat. A warm server measures carry-over from the previous configuration, not the configuration.
Token counts come from the server's own tokenizer via apply_template+tokenize, not from an estimate, and the counting method is recorded in the output.
Then the mechanism. parse_slot_log.py reads llama-server's log lines directly:
slot get_availabl: id 0 | task 12 | selected slot by LCP similarity,
f_sim_best = 0.716 (> 0.100 thold), f_keep = 1.000
slot print_timing: id 0 | task 12 | prompt eval time = 4238.90 ms /
220 tokens (19.27 ms per token, 51.90 tokens per second)
Those two lines separate the scheduler's decision from the hardware's price:
$$\text{latency lost} = \frac{\text{tokens recomputed}}{\text{prefill throughput}}$$
| host | tokens | prefill rate | modelled | measured | error |
|---|---|---|---|---|---|
| Neoverse N2, 4 vCPU | 220 | 51.9 tok/s | 4332 ms | 4204 ms | 3.0% |
| x86, 8 threads | 220 | 98.6 tok/s | 2282 ms | 2313 ms | 1.4% |
Two numbers from the server's own log predict measured TTFT within 3% on both architectures. And the scheduler's decision is byte-identical across them — same 0.716, same 220 tokens — while the price differs 1.9x. The mechanism is architecture-independent; the cost is not.
The Arm measurements run on a free GitHub-hosted ubuntu-24.04-arm runner (Azure Cobalt 100 / Neoverse N2). Fork the repo, dispatch the bench workflow, and the full adjudicated result lands in 47 minutes for $0. The model is pinned by SHA-256 and the fetch script deletes the artifact and exits non-zero on a mismatch, so the thing you download is provably the thing I measured.
60 runs total — 30 on Neoverse N2, 30 on x86 — with all raw data and all 60 raw server logs committed. Not just conclusions.
Challenges we ran into
Latency almost gave me four false positives. Four of the six prompt shapes showed latency spreads of 1.26–2.26x across thresholds. That looks like mild sensitivity, and an earlier version of my analysis reported exactly that — "optimal threshold differs across shapes," listing six values. Then I checked recomputed tokens: byte-identical at every threshold, across all five repetitions. The latency spread was machine noise. I rewrote the analyzer to decide sensitivity on recomputation — the causal quantity — rather than latency, the noisy consequence. Four false positives went away.
A better instrument produced a worse-sounding answer. The question "is multi-tenant back at single-tenant parity after the fix?" returns REJECTED on Arm: the fix recovers 10.5x of a 10.9x regression, leaving a real 4% residual gap. My x86 laptop, at 4.89% median RSD, had overlapping intervals and could not see it. The Neoverse N2 instance, at 0.41% median RSD, resolved it. That row stays in the table, marked REJECTED, because deleting it would have been the only way to get a cleaner story.
I found two invented numbers in my own documentation. Two similarity values in the README and the slide deck were plausible and were in no log — they had crept in as illustration and hardened into fact. I audited every numeric claim in every document against the committed artifacts and replaced them with measured values. It is the kind of error that survives review precisely because it looks correct.
One result I still cannot explain, and it is published. prefix20 at threshold 0.1 reports a chosen-slot similarity of 0.169 — a foreign slot — while recomputing only 17–19 tokens per warm request. Under \( f_{sim} \approx \frac{\text{shared prefix}}{\text{total prompt}} \) those two facts should not coexist. The measurement is deterministic across all five repeats and stands. The explanation does not, and it is in docs/results.md under a heading that says "Unexplained."
Earlier, my workload was flattering itself. All tenants sent identical turn text, which inflated inter-tenant similarity to 0.906 and exaggerated the regression — in my favour. I threw the run away and rewrote the tenants with per-tenant itineraries. The honest number is 0.716.
Accomplishments that we're proud of
The defect is structurally invisible to standard benchmarking. Every llama.cpp benchmark I could find measures one conversation. A single-conversation benchmark has no second tenant to be confused with, so it cannot see this class of defect at all. That is how a widely used server ships it unreported.
The consequence runs backwards from everyone's intuition. The larger your system prompt and tool schemas, the worse the default behaves. More shared context raises inter-tenant similarity, so a fixed low threshold separates tenants less well. Every instinct says more caching should help more. Agent workloads have the largest shared preambles of anything anyone is building — this is a foot-gun sitting directly on the path the industry is walking.
A prediction made before the data held on a workload that didn't exist yet. "The cliff sits at the workload's own similarity" was written into workloads.py, then confirmed on prefix65, a shape created after the prediction.
The repository shows the failures. Three dead hypotheses are in the git history, not edited out. The probe that killed two weeks of planned work is a committed tool with its output in the results. REJECTED and UNCERTAIN verdicts appear in the headline table next to the VERIFIED ones.
Total cost to verify every number in this submission: $0 and 47 minutes.
What we learned
Measure the opportunity before building the solution. A 60-line probe run in 45 minutes killed two weeks of work. That is the single highest-leverage thing I did on this project, and it produced no code that shipped.
Decide on the causal quantity, not the observable one. Recomputed tokens is what the scheduler actually changes; latency is that quantity multiplied by hardware and buried in noise. Ranking latency medians would have shipped four false findings with real-looking numbers attached.
Write the adjudication rules before the data exists. My hypothesis changed three times. skeptic.py did not. If I had chosen the thresholds after seeing the numbers, I would never have known whether I chose them honestly, and neither would anyone else.
A defect can be architecture-independent while its cost is not. The same scheduler decision — 0.716, 220 tokens — on both machines, priced 1.9x apart. Separating "what the software decided" from "what the hardware charged" is what made the result portable instead of anecdotal.
Precision cuts both ways. The quiet Arm instance let me resolve a real 4% gap, and that gap made my result look worse. That is what a measurement is for.
What's next for neoverse-tune — workload-sensitivity for llama.cpp on Arm
Report it upstream. The finding, the reproduction, and the log evidence go to the llama.cpp project — written personally, since their contributing guidelines prohibit AI-written posts. The most defensible proposal is not a new default but a startup warning when measured inter-tenant similarity exceeds the configured threshold, since the server already computes both numbers.
Separate microarchitecture from core count. The N2 host has 4 vCPU and the x86 host 8 threads, so the cost model's inputs are confounded. The clean experiment is Neoverse N1 (Graviton2, dotprod only) against Neoverse N2 (Cobalt 100, i8mm) at matched thread count. The harness already runs on both.
Add concurrency. Every measurement here is strictly sequential, which isolates slot selection from queueing and deletes the dominant real-world variable. This is the largest open gap and it is stated as such in docs/results.md.
Find where the recommendation fails. No workload was found where raising the threshold hurts. One probably exists — chat with short prefixes and many users is the obvious candidate — and I did not look for it. A recommendation with no known failure case has not been tested hard enough.
Scale the preamble. The central claim is about preamble size, and exactly one value (550 tokens) was tested. Sweeping it converts a demonstrated effect into a curve you can predict from.
Log in or sign up for Devpost to join the conversation.