TuneForge

An agent that tunes AI inference on Arm cloud CPUs — and rejects its own wins when they cost accuracy. Up to ×3.08 measured on Graviton4; every recipe replays on your host.

Comment

TuneForge

Inspiration

Arm cloud CPUs are the most cost-efficient way to serve small AI models, and almost nobody tunes them. The knob space — compile target, micro-kernel library, quantization, threading, affinity — is worth 1.7× to 3.1× throughput. But hand-tuning costs an engineer-day per model × instance combination, and the folklore ("always enable X") goes stale faster than the blog posts documenting it.

We wanted the tuning engineer to be software: an agent that reads hardware evidence the way a Neoverse performance engineer does, runs disciplined experiments, and ships results anyone can verify.

The obvious build was "give an LLM some tools and let it loop." We rejected that on day one — and that decision is the whole project.

What it does

TuneForge is an agent-driven inference optimizer for Arm Neoverse cloud CPUs. Point it at a model on a Graviton or Axion instance. It classifies the bottleneck from live PMU counters using Arm's own topdown methodology, changes exactly one knob, measures, gates on model quality, and repeats — then emits a replayable recipe and an honest report.

tuneforge run ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf --strategy agent --baseline generic
  → recipe.yaml    winning config + every accepted step + the hypothesis behind it
  → report.md      step-by-step narrative, hardware evidence, Pareto frontier, caveats
  → charts + steps.jsonl    the full audit trail

Measured, single sessions, artifacts committed in the repo:

workload engine baseline → tuned result
Llama 3.2 3B Q4_0 llama.cpp 46.66 → 85.17 t/s ×1.83 generation, ×3.75 prompt, TTFT −73%, $/Mtok −45%
Qwen2.5 7B Q4_0 llama.cpp 21.66 → 45.45 t/s ×2.10
Llama 3.1 8B Q4_K_M llama.cpp 21.48 → 36.61 t/s ×1.70
ResNet50 ONNX Runtime 51.76 → 109.35 img/s ×2.11 (int8, accuracy-gated)
DistilBERT SST-2 ONNX Runtime 62.52 → 192.76 seq/s ×3.08 (int8-dynamic, accuracy 90.5%)
MobileNetV4-small ONNX Runtime 754.71 img/s ×1.00 — int8 was +147% and the gate rejected it

Portability is a number, not a slogan. A recipe tuned on GCP Axion replays on AWS Graviton4 within +2.2% — same Neoverse V2 host class, different cloud, different hypervisor.

Try it yourself on Arm64 — ~16 measured minutes from a fresh instance

Verified on AWS Graviton4 (c8g.4xlarge) and GCP Axion (c4a-standard-16), Ubuntu 24.04.

git clone https://github.com/edenfunf/tuneforge && cd tuneforge

bash scripts/setup_graviton.sh      # toolchain + llama.cpp + model + quality dataset
source .venv/bin/activate

tuneforge env       # expect: cpu_model = "Arm Neoverse V2"
tuneforge knobs     # knob catalog + per-host availability

tuneforge run --demo                # 5-minute end-to-end demo, real pipeline

tuneforge run ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf \
  --strategy agent --baseline generic \
  --quality-dataset ~/datasets/wikitext-2-raw/wiki.test.raw

# VALIDATE any shipped claim on your own host, in one command:
tuneforge apply recipes/llama-3.2-3b-q4_0-aws-graviton4-c8g4xl.yaml
#   → tg_tps: recipe 85.17 vs replay <yours> (±x.x%) → WITHIN TOLERANCE

ONNX Runtime track: bash scripts/setup_ort.sh, then tuneforge run <model>.onnx --engine ort --calibration-data val.npz.

Off-Arm (macOS/Windows/x86) everything except benchmarking runs, so the tests and the knob catalog are inspectable anywhere: python -m pytest -q → 186 passing.

Guided walkthrough with expected output at every stage: docs/learning-path.md.

How we built it

Three layers, each doing only what it is best at.

layer owner why
Analysis — classify the bottleneck deterministic code Arm's topdown method is a published decision tree. A decision tree belongs in auditable code, not in a model's latent space. Cross-checked against Arm's own topdown-tool: backend-bound within 0.4 percentage points.
Experiment design — which knob, which value, what will happen Claude The part that genuinely needs judgment: reading evidence to order the search, and knowing when to stop.
Judgment — accept, reject, grade deterministic code Single-variable enforcement, budgets, noise guards, the quality gate, and a prediction grader the model cannot influence.

Every proposal carries a falsifiable prediction (one metric, one direction, and optionally which topdown bucket it claims to move). The orchestrator grades it after measurement — the model never marks its own homework — and the running scorecard is fed back into the next prompt.

Guarantees are properties of the code, not behaviours we hope a model exhibits. An API outage degrades the session to deterministic coordinate descent; it can never corrupt a result.

Challenges we ran into

  • The flagship cloud had no counters. GCP Axion guests expose no CPU PMU at all (No ACPI PMU IRQ — not even cycles). Every hardware-evidence claim had to move to AWS Graviton4. We turned the constraint into the cross-cloud story: tune on one, replay and verify on the other.
  • Graviton4's virtualized PMU has only two general-purpose counters. Arm's own topdown-tool could not compute Retiring or Bad-speculation on it. Our multiplexing-tolerant derivation completed the set — and still landed within 0.4pp of the tool on the buckets it could compute.
  • Our CNN quantization recipe silently destroyed a transformer. Static per-channel int8 on DistilBERT scored 49% — random guessing — with NaN calibration scales, because activation outliers collapsed the range. That is not an honest tradeoff, it is a recipe that does not transfer. We added an int8-dynamic lever; the agent then proposed it itself and the gate confirmed +208% at 90.5% accuracy.
  • The host moves under you. We measured the same configuration 11.5% apart across two sessions on one instance. Naive tuning absorbs that wander into its claims. So the session re-measures its baseline at the end (drift check) and re-measures a stale reference before any spaced-out acceptance.
  • We audited our own tool days before submission and found two correctness bugs. Loading an ONNX recipe as llama.cpp silently succeeded, dropping the quantization setting and inventing knob values that were never measured; and the session baseline was cache-eligible, so a stale number could become the denominator of every claim. Both fixed, both with regression tests, all documented in docs/design/audit-2026-08-14.md.

Accomplishments that we're proud of

The tool argues with you.

  • It refused a +147% win. int8 quantization on MobileNetV4 more than doubled throughput. Top-1 accuracy fell 3.6 points (−4.6% relative) against a 3.0% limit, so the gate rejected it and the shipped recipe is the slower fp32 configuration. On ResNet50 the same lever was rejected at −3.3% relative, then accepted at −1.3% once quantization was done per-channel — the gate taught us the right recipe on the way.
  • The prediction record is graded, not claimed. Across all 16 archived on-hardware sessions: 18 correct, 1 wrong, 9 inconclusive (inside the noise band, honestly marked). Eleven of the correct ones were on steps that were rejected — the agent designed disconfirming probes and called them right.
  • We went looking for KleidiAI's win and published the opposite. Four hosts, two quantization formats: upstream ggml's repack kernels were ahead every time on current llama.cpp. Both paths are Arm engineering; the tool measures instead of assuming, and our lab notes retract our own earlier findings when the data says so.
  • The extensibility claim has a receipt. We added a second engine (ONNX Runtime) during the contest. The commit that implements the engine itself touched zero core files.
  • Arm Performix evidence ships inside the report: on the portable baseline, 85% of samples sit in one scalar quantized-matmul kernel — Arm's own tooling showing exactly what the +83% build-tier fix fixes.

What we learned

The honesty machinery is the product. Every guard we built — prediction grading, drift checks, direction-aware quality gates, the stale-incumbent refresh — caught something real within days of existing. One apparent +4.5% was actually +1.0% of tuning and 3.5% of host drift.

Efficiency-minded design on Arm64 is not just picking fast kernels. It is knowing which of your wins are real. A benchmark number without a gate, a drift check, and an attribution trail is a claim, not a measurement — and claims do not survive contact with someone else's machine.

We also learned that the strongest thing an optimizer can do is say no. A quality gate that never rejects is not a gate.

What's next for TuneForge

Edge is the same loop with bigger stakes: thousands of device SKUs, big.LITTLE placement, thermal throttling as first-class drift, NPU delegates — where per-device automated tuning stops being a convenience and becomes the only option. The adapter contract is ready and we already proved its cost.

Nearer term: calibrate the bottleneck classifier's memory-pressure thresholds against real hardware (our audit showed the current one can never fire on Graviton4's counter set), grade predictions on mechanism as well as outcome, and grow the recipe library — because every published recipe makes the next agent session smarter. Coordinate descent needed ten steps to certify what the agent, primed with three prior recipes, found in three.

Submission-form fields

  • Project name: TuneForge
  • Elevator pitch: An agent that tunes AI inference on Arm cloud CPUs — and rejects its own wins when they cost accuracy. Up to ×3.08 measured on Graviton4; every recipe replays on your host.
  • Try it out: https://github.com/edenfunf/tuneforge
  • Video demo link: [YouTube URL]
  • Track: Cloud AI
  • License: MIT (visible in the repository About section)
  • Built with: python · llama.cpp · onnx-runtime · arm-performix · arm-neoverse · aws-graviton · gcp-axion · linux-perf · claude · anthropic-api · pydantic · typer · rich · numpy · matplotlib · pytest · github-actions · asciinema · yaml · int8-quantization · kleidiai

Built With

  • anthropic-api
  • arm-neoverse
  • arm-performix
  • asciinema
  • aws-graviton
  • claude
  • gcp-axion
  • github-actions
  • int8-quantization
  • kleidiai
  • linux-perf
  • llama.cpp
  • matplotlib
  • numpy
  • onnx-runtime
  • pydantic
  • pytest
  • python
  • rich
  • typer
  • yaml
Share this project:

Updates

Submission history