coldpath: verify your Arm binary uses the matrix hardware

Portable Arm AI builds ship with the matrix unit off: 5.75x slower prefill on a Neoverse N2 cloud CPU. coldpath audits any shipped binary, caught the top offender (Ollama), and gates it in CI.

Comment

Challenge track: Cloud AI (Arm64 cloud inference performance + a production-ready developer workflow: a CI gate that keeps cold builds off your Arm fleet).

Project Overview

coldpath is a linter and CI gate that proves whether an Arm binary actually uses the chip's matrix hardware. Pointed at the ecosystem, it found that Ollama's official Windows-on-Arm build (the most popular local LLM runner) executes zero matrix or dot-product instructions, filed a one-line upstream fix worth 5.75x on prompt processing measured live on Azure Cobalt 100 (Neoverse N2), and ships as a CI gate so the regression can't come back. It should win because it clears the line between "runs on Arm" and "optimized for Arm" from both sides: it measures the gap and closes one, on real cloud Arm silicon, with every claim reproducible from the repo.

Functionality / Output

Point coldpath at any AArch64 binary, directory, wheel, APK, or archive (ELF, PE, Mach-O) and it returns a verdict, HOT (SME) / WARM (i8mm or bf16 matrix) / TEPID (dot-product only) / COLD (none) / UNKNOWN, with per-feature instruction counts, decode coverage, and a sha256 for pinning. --require i8mm exits non-zero, so it fails a pull request that would ship a cold binary. The outputs are: the CLI verdict and JSON report, the reusable GitHub Action, the upstream fix (Ollama PR #17654), a scan scoreboard of official Arm AI binaries (RESULTS.md), and a learning-ready field guide (docs/GUIDE.md).

Setup Instructions (build / run / validate on Arm64)

Run it on any OS, no Arm hardware needed (it analyzes Arm binaries):

pip install coldpath
curl -LO https://github.com/ollama/ollama/releases/download/v0.31.2/ollama-windows-arm64.zip
coldpath ollama-windows-arm64.zip          # -> COLD, 0 matrix instructions

On an Arm64 machine (Graviton, Cobalt, Axion, a Pi, Apple Silicon), also check your own deployment:

coldpath "$(python -c 'import llama_cpp,pathlib;print(pathlib.Path(llama_cpp.__file__).parent)')"

Validate the tool: pip install -e ".[dev]" && pytest (26 tests); python scripts/verify_ladder.py <llama.cpp release dir> reproduces the ISA ladder; fork the repo and run the "What the cold path costs" workflow to reproduce the cost benchmark on the free Neoverse N2 runner.

Inspiration

I wanted to know something basic about my own setup: when I run a quantized LLM on an Arm machine, is it actually using the chip's matrix-multiply instructions, or silently falling back to scalar code? I could not find a way to answer it. Arm's matrix kernels (i8mm, SME2, dot-product) are selected at compile time, so a wrong build flag does not make them slow, it removes them from the binary entirely, and nothing warns you. The only advice on offer was to attach a profiler to real Arm silicon and read assembly symbol names. That gap is the whole project: a question a competent engineer cannot answer today, made answerable.

What it does

coldpath disassembles any AArch64 binary (ELF, PE, Mach-O; also inside pip wheels, APKs, and tarballs) and reports whether it contains the chip's matrix and dot-product instructions: SME/SME2, i8mm (smmla), bf16 matmul (bfmmla), and dotprod (sdot). Absence is the airtight direction and it is the whole finding: zero smmla means no core can run an i8mm matmul, however the library dispatches at runtime. For ggml that is also decisive for what executes, because ggml selects these kernels at compile time. It needs no Arm hardware and no profiler, and it audits binaries you did not build (a shipped release, a pip wheel, an APK), not just your own tree. It runs as a CLI and as a CI gate that fails a pull request which would ship a "cold" binary to an Arm cloud fleet.

The cost is real and cloud-native. On Azure Cobalt 100 (Neoverse N2), a cold build (-march=armv8-a, no matrix/dot-product kernels) runs prompt processing at ~95 tok/s; the one-flag fix runs at ~545, a 5.75x prefill speedup (and ~2.3x on generation), about $0.45 to $0.08 per million prompt tokens, measured live on the free GitHub Arm64 runner. Cold is a distribution hazard, not a build default: a native build detects the host and comes out warm, but portable and cross-compiled binaries (how prebuilts are made) must pin -march or fall back to baseline armv8-a, which is exactly what Ollama's Windows build did.

Pointing coldpath at the ecosystem found the most striking instance: Ollama's official Windows-on-Arm build executes zero matrix and zero dot-product instructions, while llama.cpp from the same ggml source on the same OS ships them. Ollama just builds it with the missing flag. I filed the one-line fix upstream (PR #17654).

How I built it

The core is a capstone-based disassembler that classifies instructions by their register operands (ZA tiles for SME, Z/P registers for SVE, V registers for NEON), because capstone leaves its instruction-group metadata empty for SVE and SME, so the obvious approach silently reports zero. The sweep resyncs on a 4-byte grid, which is provably sound on AArch64's fixed-width instructions where it would not be on x86. I validated it against ground truth I did not author: llama.cpp's own release ships eight CPU backends with the ISA level in each filename, and coldpath reproduces that ladder exactly, with ONNX Runtime as a positive control proving a "zero" means a real absence. To measure the cost, I built llama.cpp three times on a free GitHub Arm64 runner (Neoverse N2), changing only -march, scanned each build with coldpath, and benchmarked each with llama-bench.

Challenges I ran into

The hardest part was making "absence is proof" bulletproof. An adversarial review of my own claims found that the disassembly could, in principle, stop at a literal pool embedded in code and report a false "cold." I fixed it with the resyncing sweep and a coverage gate that refuses to assert absence it cannot back up, and I turned the objection into a stated strength (AArch64 fixed-width makes it sound). The second challenge was scope: on Apple Silicon, matmul can route through the Accelerate/AMX unit, which a disassembler cannot see, so I scoped the headline to Linux/Neoverse, Windows-on-Arm, and Android, where the ISA path is the only path.

Accomplishments that I'm proud of

I did not just diagnose a problem, I fixed it and measured it. On a Neoverse N2, the safe dot-product fix I filed upstream restores the sdot kernels on every dot-product-capable Arm core and recovers 5.75x on prompt processing (95 to 545 tok/s), i8mm reaches 6.9x where the silicon supports it, and generation improves ~2.3x. I did not even need to rebuild Ollama to prove the flag is the cause: llama.cpp ships the same ggml source built with the flag, and coldpath scores it WARM where Ollama is COLD, two shipped binaries one flag apart. Every number regenerates from CI on free Arm hardware, so a judge can re-run it rather than trust a screenshot. And the tool found a real defect in software millions of people run, on its first day.

What I learned

How deep the gap is between "supports Arm" and "fast on Arm." The same KleidiAI dependency is shipped by ONNX Runtime with SME2 on by default and by ggml with it off by default, so identical hardware gets opposite results purely from a build flag. I also learned the exact mechanics of ggml's compile-time ISA dispatch and why static analysis is the right tool for auditing it.

What's next for coldpath

Distinguishing SME from SME2, detecting runtime-gated dispatch (present-but-not-selected) as a third state, and growing the ecosystem scan into a public dashboard of which shipped Arm AI binaries are hot or cold. And landing the Ollama fix.

Built With

  • aarch64
  • arm64
  • azure-cobalt-100
  • capstone
  • dotprod
  • executorch
  • ggml
  • github-actions
  • i8mm
  • kleidiai
  • llama-bench
  • llama.cpp
  • mit-license
  • neoverse
  • ollama
  • onnx-runtime
  • pefile
  • pyelftools
  • python
  • sme2
Share this project:

Updates

Submission history