Armsmith

Armsmith finds why your AI repo is slow on Arm and drafts the fix PR — 7.4× measured on Neoverse-N2, and it deletes the fixes it can't prove.

Comment

Inspiration

A solo founder's cloud bill doubled overnight when her RAG service outgrew its instance. The Graviton migration guide is 40 pages of compiler flags she doesn't understand, and her next invoice lands in nine days. Arm64 cloud (Graviton, Ampere, Axion) runs AI inference 20–40% cheaper per unit of throughput — but capturing that needs performance-engineering lore almost nobody has: image architecture pins, -mcpu/-march flags, BLAS backend choice, dtype discipline, quant formats only some ISAs can runtime-repack, thread topology vs. vCPU count. Teams either never migrate, or migrate and silently leave 2–3× on the table. Arm's own Learning Paths teach a human to fix this by hand; nothing automated the loop. So we built the agent that does — and, crucially, one that refuses to lie about its results.

What it does

Armsmith is a CLI agent that profiles an AI repo on Arm, diagnoses why it's slow on aarch64 with a 13-rule anti-pattern pack, and drafts a pull request whose every fix has passed a reproduce-benchmark gate — median-of-N, scaled-MAD noise bands, output-hash equality. (PR rendering ships today and is dry-run only — see step 5; posting is on the roadmap and marked as such in the code rather than implied here.)

The trust inversion is the whole point: the planner proposes fixes; only the reproduce gate can claim a result. (Today's planner is a deterministic priority sort — the Claude tool-use loop is contract-pinned and raises rather than pretending. That is deliberate: the gate is what makes a result trustworthy, so nothing about the thesis depends on the planner being an LLM.) Fixes that don't clear the noise band are deleted by the tool itself and reported, never hidden. armsmith verify re-derives every statistic in a report from its embedded raw samples — change one digit and it prints VERIFY FAILED. A judge never has to trust a screenshot.

The hero flow (steps 1–6 and 8 run offline on any x86 laptop, zero Arm hardware; step 7 is the live leg and needs aarch64 — a free GitHub arm64 runner is enough):

  1. armsmith scan <repo> — static aarch64 anti-patterns (R1 amd64-pin, R4 float64 coercion, R12 CI-matrix) on a real clone, no hardware.
  2. armsmith diagnose --replay <bundle> — baseline bench + 13-rule scan + the reproduce gate → a signed report.json.
  3. armsmith witness <before> <after> — before/after objdump count of SDOT/UDOT/SMMLA/USMMLA (e.g. 0 → 4 dotprod): deterministic proof the Arm dot-product / int8-matmul kernels are actually emitted. Not a stopwatch — machine code.
  4. armsmith verify <report> — recompute every stat + gate verdict from raw samples; tamper → fail.
  5. armsmith pr <report> — the bot PR: one commit per surviving fix, an evidence table | metric | before | after | Δ | noise band | PMU Δ |, and a dropped-with-reasons log. Dry-run / render-only today — it prints exactly what would ship and makes no network call (ghpr.py has no network code path at all, and real PR posting is an unchecked box on the roadmap). We would rather ship the rendering and say so than claim an integration we haven't earned; that is the same rule the gate applies to performance numbers.
  6. armsmith ci --replay <bundle> — the same gate as an exit-code GitHub Action for runs-on: ubuntu-24.04-arm.
  7. armsmith bench-live — the live leg on real Arm silicon: compiles one kernel two ways, disassembles both, ABAB-interleaves the timed runs, and puts the samples through that same gate. Refuses to run on anything that isn't aarch64 — there is no code path that yields an Arm number off Arm hardware, and a unit test asserts it.
  8. armsmith rules export — 13 x86→Arm migration-template cards, each linking the Arm Learning Path that teaches the fix by hand.

Final output / the honest number — measured, not estimated. armsmith bench-live runs the whole gate on real Arm silicon. On a GitHub-hosted ubuntu-24.04-arm runner (Neoverse-N2, gcc 13.3.0) it compiles one int8 dot-product source twice, differing only in the -march flag rule R2 exists to flag, and measures both:

-O3 -march=armv8-a -O3 -march=armv8.2-a+dotprod
SDOT in the hot symbol 0 1
median kernel time 0.059975 s 0.008123 s
Δ — −86.5% — a 7.4× speedup
noise band (k=3) — ±0.000144 s
output hash — identical
gate verdict — keep

Read the first row before the timings: ARMv8.0 has no dot-product instruction, so the baseline cannot contain one. The stopwatch says 7.4×; the disassembly says why. CI re-runs this on every push, verifies the ed25519-signed report, and uploads it as a downloadable artifact — so the number is re-derivable by anyone with the repo and an Arm runner.

And on the ragserve seed, the same gate turns 6 candidate fixes → 4 kept, 2 dropped (1 inside the noise band, 1 output-hash mismatch). The tool deleting its own unprovable fixes is the point.

How we built it

Layer Technology Why
CLI / core Python 3.11–3.12, Typer, Rich Fast to ship, arch-clean, installs identically on aarch64
Statistics custom benchstats (median-of-N, scaled-MAD, noise band, p50/p95) The gate needs defensible, reproducible math — not a black box
Trust chain cryptography (ed25519), canonical-JSON SHA-256, jsonschema Tamper-evident, independently re-verifiable reports
Arm targeting lscpu/getauxval ISA fingerprint (dotprod/i8mm/SVE/SVE2/BF16/SME), GCC -mcpu/-march, llama.cpp/KleidiAI, perf/PMU This is the actual optimization surface — the product is Arm
Live Arm bench GCC + binutils objdump on aarch64, driven by armsmith.livebench Compile the A/B from one source, then count the SDOT in the disassembly — instruction-level proof, not a stopwatch
CI GitHub Actions on native ubuntu-24.04-arm + ubuntu-22.04-arm + x86 matrix, plus a dedicated live-bench job on Neoverse-N2 Proves the package is arch-clean and takes a real measurement on real Arm silicon, every push

Quality & Security Engineering (production-maturity floor, not the headline): 457 passing tests at 100% line coverage (2,746 statements, 0 missed), fully offline; ruff lint gate; mypy (advisory); a security-gated CI pipeline with CodeQL (SAST), Dependabot (SCA), and TruffleHog secret scanning; a JSON-Schema check on the report schema; and an end-to-end offline loop (scripts/verify_offline.py). 100% GitHub community profile (Code of Conduct, Contributing, Security policy, issue/PR templates).

Why Arm (the exclusivity defense): Armsmith isn't an app that happens to run on Arm — every rule, probe, and fix is aarch64-specific. It fingerprints Arm ISA extensions to route rules; its fixes are -mcpu=native/KleidiAI/GGUF-repack/thread-topology moves; its witness proof counts Arm dot-product and i8mm-matmul instructions in real machine code; its CI takes a real measurement on a Neoverse-N2 Arm runner. The headline result is literally an Arm ISA level being switched on — on x86 there is no SDOT to count and nothing here has meaning. Remove Arm and there is no product.

Challenges we ran into

  • Making an LLM agent trustworthy. An agent that claims speedups is worthless if you can't check it. We inverted the architecture: the planner only proposes; a deterministic reproduce gate (median-of-N + MAD noise band + output-hash equality) is the only thing that can claim a win, and verify recomputes every number from embedded raw samples so tampering is detectable.
  • Proving Arm optimization with zero Graviton hardware in-hand. We first built two hardware-free proofs — verify's stat recompute, and witness, a before/after objdump count of SDOT/UDOT/SMMLA/USMMLA — both runnable on any x86 laptop. Then we realised the free ubuntu-24.04-arm hosted runners are Graviton-class Arm silicon, and turned the CI substrate itself into the measurement rig. No instance to rent, no credentials, and the result reproduces on every push.
  • Getting a compiler to prove the point. The kernel had to be noinline and externally linked: as a static function GCC's IPA pass cloned it to dot_i8.constprop.0, and the witness could no longer find the symbol to disassemble. We also could not validate any of this on the dev machine — Apple clang silently ignores -march on arm64 (M-series is already ARMv8.5), so both builds came out identical locally. The CI runner was the only honest oracle, so we iterated against it.
  • Honest fixtures. Every replay bundle is labeled "synthetic": true and refused by the loaders when unlabeled, so no synthetic number can ever masquerade as a hardware measurement.

Accomplishments that we're proud of

  • The "agent proposes, silicon disposes" trust model — the tool visibly deletes 2 of its own 6 fixes because it couldn't prove them, and the counter reads proposed 6 · proven 4 · refused 2 · faked 0.
  • A measured 7.4× on real Arm silicon with instruction-level proof — SDOT 0 → 1, −86.5% against a ±0.24% noise band, output-hash equal, gate verdict keep, ed25519-signed and re-verified in CI. Reproduced across independent runs (−86.46% / −86.37%).
  • A 457-test, 100%-line-coverage, security-gated, arm64+x86 CI harness — a real engineering product, not a weekend toy.
  • 13 reusable x86→Arm migration templates, 10 linked to the Arm Learning Paths the judges authored.
  • A tamper-evident, publicly-schema'd signed-report format anyone can build a viewer or CI gate against.

What we learned

  • In an "optimization" hackathon, the honest, reproducible number beats the bigger unverifiable one. Calibrated honesty (synthetic-vs-live labels on every number, dropped fixes shown) is a feature, not a weakness.
  • Free CI runners are real silicon. We spent a long time treating "no Graviton box" as the blocker, when the arm64 hosted runner already in our CI was a Neoverse-N2 we could measure on. The constraint was imaginary; the measurement took an afternoon.
  • Arm-specific performance work has a small number of high-leverage anti-patterns; encoding them as data (YAML rules + Learning-Path citations) makes the knowledge reusable beyond our CLI.

Known limitations (calibrated honesty): LiveProbe over SSH (--target ssh://…), the Claude planner tool-use loop, live PR posting, perf/PMU + Performix + llama-bench ingestion, and cosign-in-CI are all TODO(S1) and marked as such in code. The live measurement above is one microbenchmark on one runner — it proves the gate works on real silicon, and is deliberately not a claim about anyone's model. Every other number in the repo comes from a replay bundle labeled "synthetic": true; live reports carry "mode": "live", "synthetic": false. Every number says which it is.

What's next for Armsmith

  • Extend the live leg from a microbenchmark to a real workload: ragserve end-to-end throughput on Graviton with a real PR and a cost receipt.
  • LiveProbe driving real perf/hyperfine/llama-bench over SSH; the Claude planner loop.
  • cosign/Sigstore keyless attestation of report.json in CI; a formal GitHub Marketplace listing for the armsmith action.

Setup Instructions (Devpost "Setup Instructions — Arm64")

Nothing to install (~10 s) — Armsmith is on PyPI, published from CI by Trusted Publishing:

uvx armsmith scan .        # or: pipx install armsmith  ·  pip install armsmith

Record a bundle from your own machine and diagnose your own repo — the probe rules run on your code, not on our fixtures. record captures what your host can honestly report and copies in any real instrument output you already have; anything it cannot observe is omitted, never guessed:

armsmith record . --out ./bundle --python .venv/bin/python
armsmith diagnose --replay ./bundle

The manifest it writes declares "synthetic": false, and env/proc_maps are never captured — a bundle is a published artifact, and those carry CI tokens and host paths.

Zero-hardware validation (~2 min, any x86 laptop — the primary judge path):

git clone https://github.com/edycutjong/armsmith && cd armsmith
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
python -m pytest -q                                             # 457 passing, 100% cov, offline
armsmith scan fixtures/replays/scenario_ragserve
armsmith diagnose --replay fixtures/replays/scenario_ragserve
armsmith witness fixtures/witness/objdump_before.txt fixtures/witness/objdump_after.txt
armsmith verify fixtures/replays/scenario_ragserve/report.json  # -> VERIFY OK
armsmith ci --replay fixtures/replays/scenario_ragserve         # -> CI GATE PASSED
python scripts/verify_offline.py                                # -> ALL CHECKS PASSED

Tamper proof: edit one digit in a samples array of fixtures/replays/scenario_ragserve/report.json, re-run armsmith verify … → VERIFY FAILED.

On native Arm64 (Graviton c7g/c8g, Ampere, Axion, or a GitHub ubuntu-24.04-arm runner): the same commands run identically, armsmith doctor --offline --replay … shows the host's dotprod/i8mm/SVE routing, and you can take the real measurement yourself:

armsmith bench-live --require-witness   # compiles the A/B, counts SDOT, measures, gates, signs
armsmith verify report-live.json        # -> VERIFY OK (recomputes every stat from raw samples)

bench-live refuses to run on non-aarch64 hosts. Full live capture against your own workload (perf/hyperfine/llama-bench over SSH) is the TODO(S1) path; Armsmith never fabricates a hardware number.

How this was significantly updated during the Submission Period (rules §4 "New & Existing")

The offline core was built 2026-07-04/05 — inside the Submission Period (Jun 10 – Aug 14 2026). During this cycle (2026-08-03 onward) it was significantly updated:

  • The live Arm reproduce gate (2026-08-08) — the largest addition. New armsmith bench-live + armsmith.livebench + bench/int8_dot.c: the first code path in the project that produces a hardware measurement. LiveProbe went from a stub to a working local-exec backend. A new native arm64 CI job takes the measurement on every push, verifies the signed report, and uploads it. This is what produced the SDOT 0 → 1 / −86.5% result above.
  • Engine ported into the public build/ repo; hero-flow commands scan/witness/pr/ci registered.
  • rules export migration cards + the Arm Learning-Path citations added.
  • Security-gated CI + community health files added; CodeQL green with 0 alerts on the public flip.
  • Suite grown to 384 passing at 100% line coverage mid-cycle (from 219), then to 457 with the record work below; and the packaged wheel is now built and exercised from a clean venv in CI — which caught the report schema never shipping inside it.
  • Rebranded to Armsmith with a full asset suite; landing page + pitch deck now deploy straight from site/ in this repo, so the page a judge loads is the source they can read.
  • armsmith record (2026-08-09) — the tool stopped being usable only on our own data. Every probe-backed rule reads from a bundle, and until this landed no command produced one, so ten of thirteen rules only ever ran against fixtures shipped in this repo. record captures what a host can honestly report and ingests real instrument output the operator already has, writing a bundle that declares "synthetic": false. It also fixed a provenance bug it exposed: reports built from genuinely recorded data were being stamped synthetic, because provenance was derived from transport rather than tracked separately.
  • Published to PyPI (2026-08-09) via Trusted Publishing (OIDC, no token in the repo), so adoption is uvx armsmith scan . rather than a clone; sdist + wheel are attached to every GitHub Release.
  • R4 precision fixed against a real repo (2026-08-09). On a fresh clone of huggingface/text-generation-inference the rule produced 5 findings, 4 of them false — integer permutation arrays in the Marlin GPTQ path — and the patch it suggested would have corrupted an index array. It now reasons per numpy constructor: 5 findings / 4 false → 1 / 0.
  • Suite grown again to 457 passing at 100% line coverage.
  • Judge zero-hardware quickstart verified end-to-end; 2:55 demo video recorded with captions, and re-cut 2026-08-09 so scene 6 carries the real Neoverse-N2 measurement instead of a replay figure.

Built With

  • aarch64
  • arm-neoverse
  • c++
  • codeql
  • cryptography
  • dependabot
  • ed25519
  • github-actions
  • graviton
  • hyperfine
  • jsonschema
  • kleidiai
  • llama.cpp
  • neoverse-n2
  • objdump
  • perf
  • pmu
  • pytest
  • python
  • pyyaml
  • rich
  • ruff
  • sdot
  • typer
  • ubuntu-24.04-arm
Share this project:

Updates

Submission history