kv-flash-advisor

Inspiration

I wanted to build a SOC alert-triage assistant and identify ways to optimise performance. But while I was benchmarking, Claude (the tool I used to vibe code this project) discovered llama.cpp's -fa auto heuristic was silently picking the slower flash-attention setting for prefill on Arm CPU. This seemed like a potentially more broadly useful path to pursue.

What it does

kv_flash_attn_advisor.py is a diagnostic tool that measures whether llama.cpp's -fa auto default actually picked the fastest flash-attention/KV-cache setting for your model, on your hardware. It runs the default alongside forced -fa on/-fa off, applies Welch's t-test so a result isn't trusted until it's statistically distinguishable from run-to-run noise, and includes --accuracy-check to verify a faster setting doesn't quietly make your task worse. provision.py turns that into an actual deployment decision: it only writes out resolved flags if the evidence clearly supports one setting, and refuses with a specific reason otherwise. SOC Alert Triage Copilot, a small agentic pipeline that classifies security alerts, is the workload I used to validate it against something production-shaped.

The headline finding turned out to be narrower and more specific than "an Arm problem": it's tied to SVE specifically. The same test on a non-SVE Graviton2 core showed no effect at all, while SVE-capable Graviton3/4 showed the default losing by up to 258%.

How we built it

I worked with Claude throughout this project; for implementation, testing and drafting docs. I repeatedly asked it to walk through the tool step by step as if it were a fresh developer to try to break it. This led to identification and resolution of multiple bugs. To reduce the chance of hallucinations, I pushed back on findings to ensure they were validated before documenting .

Everything runs on llama.cpp built with GGML_CPU_KLEIDIAI=ON across AWS Graviton2/3/4 instances, self-quantized models (Q8_0/Q4_0) rather than using a pre-quantized download. The accuracy finding came from running the project's classification pipeline against both settings on a 10-alert test corpus with documented expected answers.

Challenges we ran into

Usually when I'm using AI tools it is to automate a task I am already doing or to build a project that very much aligns with my area of expertise. This challenge was outside my comfort zone and the main issue I ran into was being able to ensure that what Claude was building and claiming was correct e.g. it overinflated claims of how important issues it had identified would be to the developer community (which were also backed up by Gemini).

Two concrete examples: early on, a reported p-value turned out to rest on a standard deviation that was never actually measured; llama-bench had reported it as ± 0.00, below its own display precision, and a number had been substituted in to compute a significance test anyway. I caught it, we removed the fabricated figure, and made "never invent a result or its inputs" a standing rule for the rest of the project. Separately, an early Arm-vs-x86 comparison had two hidden confounds 1) a different llama.cpp build version on each side, and a different thread-to-core ratio, that could each have explained the result on their own. Both got found and eliminated by re-testing with those variables controlled before the finding was allowed to stand.

Absolute throughput on AWS shifted 2-3x between sessions on the same instance type, almost certainly because a freshly provisioned instance lands on different underlying physical hardware. That's what pushed the tool toward --profile/--save-profile and re-running at deployment time instead of hardcoding a number once.

Accomplishments that we're proud of

Building a tool outside my area of expertise.

Narrowing the finding to what the evidence actually supports, rather than the broadest version of the claim: confirmed it's specific to SVE-capable Arm cores (absent on non-SVE Graviton2), confirmed it holds across three models and two model architecture families, and confirmed that it doesn't reproduce on x86/AVX-512. Profiled it at the kernel level to find the actual mechanism (a fused attention kernel that's 2.25x slower than the naïve path on this hardware), rather than stopping at "it's slower."

Finding that the setting isn't accuracy-neutral, either. A GPU-side community report of the same symptom had existed for a while, unresolved and anecdotal (llama.cpp discussion #9646). This project put that report through the kind of test it hadn't had before, determinism-checked, verified against documented ground truth, and quantified on a real downstream task, and confirmed the effect is real on this hardware too.

The flash-attention finding is filed upstream as a data-backed report (ggml-org/llama.cpp#26559)

What we learned

Don't always trust a parameter named "auto" to choose the best setting.

When building production systems with AI, human oversight and domain knowledge is critical.

What's next for kv-flash-advisor

Two gaps we could look at: everything here was tested on 1.1–1.7B models, and the mechanism plausibly weakens at larger scale as attention becomes a smaller share of total compute. Same for hardware: every SVE-capable machine tested was AWS Graviton; other SVE implementations are an open question.

The tool's output could also be more concise, as right now it prioritizes showing its work over being skimmable.

I would appreciate any open feedback on the project.

Built With

• amazon-web-services
• arm
• c++
• claude
• graviton
• kleidiai
• llama.cpp
• python