Anchor Infer — Devpost Project Details
Project Name
Anchor Infer
Elevator Pitch
Anchor Infer is an automated Arm64 AI deployment pipeline that detects your hardware, benchmarks generic vs KleidiAI-optimized builds across model and quantization combinations, and tells you exactly which configurations actually get faster — no guesswork, no inflated claims, just measured numbers you can trust.
Inspiration
Cloud platforms like AWS and Azure are built for teams with dedicated DevOps support — but most small businesses, freelancers, and developers just have a single server and no desire to become sysadmins. We had already been building Anchor, a self-hosted deployment platform that turns any server into a one-click "private Heroku." When this hackathon's Arm Cloud AI track came up, the natural extension was obvious: AI inference deployment is one of the hardest and most opaque things for a developer to get right on their own — picking a runtime, a quantization format, and knowing whether hardware-specific acceleration actually helps. We wanted to bring the same "hide the complexity, show the truth" philosophy from Anchor to Arm-optimized AI deployment.
What it does
Anchor Infer takes a model and a target Arm64 server, and automatically:
- Detects the CPU architecture at deploy time
- Builds and runs both a generic llama.cpp inference server and a KleidiAI-optimized one, side by side
- Benchmarks both across model size, quantization level, and prompt length using a repeatable, multi-run test matrix
- Exposes the optimized model as a live, OpenAI-compatible API endpoint
- Reports a clear, evidence-backed comparison — not a single cherry-picked number, but a full matrix showing exactly which configurations benefit from Arm-specific optimization and which don't
Rather than assume KleidiAI always helps, Anchor Infer measures it, every time, and shows its work.
How we built it
The system has two parts. The agent layer is written in Go and handles architecture detection, build orchestration (compiling llama.cpp with and without KleidiAI kernels enabled), and wrapping the optimized build in llama.cpp's built-in OpenAI-compatible server mode. The benchmarking pipeline runs on GitHub Actions' free arm64-hosted runners (Cobalt 100-based), using llama-bench with a fixed, repeatable prompt set across a full test matrix: two model sizes (TinyLlama 1.1B and Mistral 7B), two quantization formats (Q4_K_M and Q8_0), and two prompt lengths (short and long), each run three times to get statistically meaningful medians and ranges rather than single noisy data points. Results are automatically written to BENCHMARKS.md and a structured test-matrix doc in the repo, so every claim we make is traceable to a real CI run.
Challenges we ran into
Our first benchmark run showed only a marginal ~2.9% generation-speed improvement from KleidiAI — far below what we expected going in. Instead of discarding the result or quietly downplaying it, we treated it as a signal to test more rigorously: we varied model size, quantization format, prompt length, and run count one variable at a time. That systematic sweep revealed the real story — KleidiAI's benefit is highly dependent on quantization format, showing a reproducible +15.6% generation speedup specifically on Q8_0 quantization (non-overlapping ranges across 3 runs), while Q4_K_M and the larger 7B model showed no meaningful gap on the hardware we had access to. Getting to that scoped, honest, statistically-grounded conclusion — instead of shipping a vague or inflated performance claim — was the hardest and most valuable part of the project.
Accomplishments that we're proud of
- Built a fully automated, reproducible benchmark pipeline running on real Arm64 hardware via free GitHub Actions runners — no cloud credit card required
- Ran a genuine multi-variable test matrix (model size × quantization × prompt length, N=3 per cell) rather than a single demo number
- Found and can technically explain a real, reproducible KleidiAI speedup (+15.6% on Q8_0 quantization) instead of assuming or overstating one
- Kept the "no-go" results in our documentation alongside the "go" result, because a submission that shows its full methodology is more credible than one that shows only its best number
What we learned
The most valuable optimization work isn't just applying an acceleration library — it's building the infrastructure to honestly measure whether it worked, under which conditions, and why. We also learned, concretely, that KleidiAI's Arm-specific kernels appear to align most closely with INT8-oriented quantization formats like Q8_0, likely because those formats map more directly onto the CPU's dot-product/matrix-multiply instructions that KleidiAI targets — while more complex packed formats like Q4_K_M may not route through the same optimized code path as cleanly.
What's next for Anchor Infer
- Test on Arm64 hardware with confirmed i8mm/SME support (e.g., Graviton4, Axion) to see if the gap widens further, since our current results are on GitHub Actions' Cobalt 100 runners
- Expand the test matrix to more model families and quantization formats to build a broader, queryable "what configuration should I use" recommendation
- Integrate this pipeline as a native deployment template inside Anchor, our existing self-hosted deployment platform, so any Anchor user can deploy an Arm-optimized, pre-benchmarked AI endpoint with the same one-click simplicity as deploying a web app
Anchor Infer — Benchmarks
This document records real, reproducible benchmark results comparing generic vs KleidiAI-optimized llama.cpp builds on Arm64 hardware. All numbers below come from actual CI runs, not estimates. Full raw run data: CI run 31775408483.
Methodology
- Hardware: GitHub Actions arm64-hosted runners (
ubuntu-24.04-arm), Cobalt 100-based, 4 vCPU - Inference engine: llama.cpp, built twice per test — once generic (no Arm-specific flags), once with
GGML_CPU_KLEIDIAI=ON - Benchmark tool:
llama-bench, fixed identical prompt set per cell - Runs per configuration: N=3, reporting median values and observed range
- Metric: tg (text generation) tokens/sec — sustained generation speed after the first token
- Variables tested: model size, quantization format, prompt length
We deliberately tested multiple configurations rather than reporting a single number, to avoid overstating or cherry-picking results.
Full Test Matrix
| Model | Quantization | Prompt length | Generic tg t/s (median) | KleidiAI tg t/s (median) | Δ% | Range (generic / KleidiAI) | Decision |
|---|---|---|---|---|---|---|---|
| TinyLlama 1.1B | Q4_K_M | short | 50.63 | 50.74 | +0.2% | overlapping | No-go |
| TinyLlama 1.1B | Q8_0 | short | 56.69 | 65.55 | +15.6% | 56.5–57.5 / 64.6–66.4 (non-overlapping) | Go (scoped) |
| Mistral 7B | Q4_K_M | short | 8.56 | 8.55 | −0.1% | overlapping | No-go |
| Mistral 7B | Q4_K_M | long | 8.63 | 8.64 | +0.0% | overlapping | No-go |
Initial single-run result (superseded)
An earlier single-run test (2026-08-14, commit 2bacf9e) showed a blended pp+tg median improvement of only +0.7% (99.77 vs 99.06 t/s), with the tg-only figure at +2.9% (50.58 → 52.04 t/s). This result was treated as insufficient evidence on its own — a single run cannot distinguish real signal from CI runner noise — and prompted the full multi-variable matrix above.
What the results show
- Bigger model alone did not help. Moving from TinyLlama (1.1B) to Mistral 7B, at the same quantization, showed no meaningful KleidiAI advantage on this hardware.
- Longer prompts did not help. Extending prompt length on the 7B model made no measurable difference to the generic-vs-KleidiAI gap.
- Quantization format is the deciding factor. Q8_0 is the only configuration tested that shows a clear, reproducible improvement — with non-overlapping ranges across all 3 runs, meaning this is real signal, not noise.
Why Q8_0 specifically
KleidiAI's Arm-optimized kernels are built around specific INT8-oriented dot-product and matrix-multiply instructions available on newer Arm CPUs. Q8_0 quantization stores weights in a format that maps more directly onto those INT8 kernel paths. Q4_K_M uses a more complex packed 4-bit format that likely doesn't route through the same optimized code path as cleanly — which would explain why it shows no measurable gain even on a larger model or longer prompts.
Honest scope of this claim
- The +15.6% result is specific to: TinyLlama 1.1B, Q8_0 quantization, short prompts, on GitHub Actions' Cobalt 100 arm64 runners.
- It is not a general "KleidiAI makes everything faster" claim — three of the four tested configurations showed no meaningful improvement.
- This result has not yet been tested on Arm64 hardware with confirmed i8mm/SME support (e.g., Graviton4, Axion), which may show a larger or different gap. This is listed as future work.
Reproducing these results
The full benchmark pipeline is defined in .github/workflows/validate.yml and can be triggered via workflow_dispatch with configurable model/quantization inputs. See docs/ci/test-matrix.md for the raw per-run data behind the medians reported above.
Log in or sign up for Devpost to join the conversation.