Inspiration
I didn't own any Arm hardware. My laptop is x86-64, and the deadline was hours away — so the first real question wasn't "what should I build" but "how do I produce Arm performance numbers that anyone can trust, on a machine that isn't Arm?"
That constraint turned out to be the most interesting part of the project. GitHub Actions gives free native Arm64 runners to public repositories. That means a benchmark can be reproducible by the judges themselves — press "Run workflow", get the numbers on real silicon, no hardware purchase, no screenshots to take on faith. Most optimization submissions ask you to believe a number pasted from someone's laptop. I wanted one you could re-run.
The target picked itself. In llama2.c's quantized runner, there is a single function carrying a comment from the author: "by far the most amount of time is spent inside this little function." It's a group-quantized int8 matmul written as a plain triple loop. On Arm, that loop is leaving whole instruction classes unused.
What it does
NeonForge replaces that function with three hand-written implementations and chooses between them at runtime:
| kernel | instruction | requires | used for |
|---|---|---|---|
scalar |
— | anything | baseline; what a stock build produces |
dotprod |
SDOT (vdotq_s32) |
Armv8.2 | decode — tokens/sec |
i8mm |
SMMLA (vmmlaq_s32) |
Armv8.6 | prefill — time-to-first-token |
One binary reads AT_HWCAP at startup and dispatches accordingly, so it stays loadable on an Armv8.0 core and still exploits SMMLA on a Neoverse server part. No separate builds, no #ifdef maze at the call site.
Alongside the kernels it ships a PyTorch-free fp32 → int8 quantizer, a benchmark harness that asserts bit-exactness rather than a tolerance, an end-to-end runner that reports prefill and decode as separate metrics, and a CI workflow that regenerates every published number on real Arm64.
Measured on Arm Neoverse-N2 (asimddp i8mm bf16 sve2, single-threaded, group size 32):
| metric | baseline | optimized | gain |
|---|---|---|---|
| prefill GEMM (768×768) | 15.95 GOP/s | 51.76 GOP/s | 3.23x |
| decode throughput | 471.94 tok/s | 562.31 tok/s | 1.19x |
| time to first token | 18.00 ms | 14.00 ms | 1.29x |
| model size on disk | 60.82 MB | 17.10 MB | 3.56x |
How I built it
I vendored run.c and runq.c from llama2.c unmodified, so the baseline is auditable rather than merely described, then wrote the kernels as separate translation units — each compiled with only the -march flags it needs, and entered only after a runtime capability check.
The scalar baseline is compiled with plain -O3 and no -march override, exactly what make runq gives you upstream. This matters more than it sounds: it would have been easy to inflate every speedup by disabling optimization on the baseline. GCC auto-vectorizes that loop to roughly 16 GOP/s on its own, and that's the number I measured against.
The SDOT kernel keeps four independent accumulators to hide the instruction's multi-cycle latency. The SMMLA kernel tiles output into 2×2 blocks — one issue computes a full 2×2 int32 tile from two 2×8 int8 operands — with odd row counts falling back to the SDOT path.
With no Arm machine to test on, I built a two-stage local verification pipeline: cross-compile with aarch64-linux-gnu-gcc, disassemble to confirm the intrinsics actually lowered to SDOT/SMMLA (64 and 15 instructions respectively, and zero in the baseline), then execute under QEMU user-mode emulation with -cpu max to check correctness. Emulated timings are worthless — QEMU translates SDOT into helper calls and reports it as slower — but emulated results are exact, which is all correctness needs.
Challenges I ran into
SMMLA does not help token generation, and saying otherwise would have been the easy lie. It's a matrix-matrix instruction: 64 MACs per issue against SDOT's 16. But decode is a matrix-vector product — there is only one activation row, so half the operand is padding and half the output is discarded. It needs at least two rows, which prefill has and decode does not. So the project routes prefill through i8mm, decode through SDOT, and reports them as separate metrics instead of blending them into one flattering tokens/sec figure. Claiming an SMMLA decode speedup is the fastest way to lose the trust of anyone who knows the ISA.
A floating-point associativity bug that looked exactly like quantization noise. My i8mm kernel multiplied by the activation scale before the weight scale; the baseline does the reverse. Integer accumulation is exact, but float multiplication is not associative, so results drifted about 1e-4 — small enough to pass for rounding error, large enough to eventually flip a sampled token. QEMU caught it because I was testing for bit-exactness rather than "close enough." A tolerance-based test would have shipped it.
An upstream correctness trap in the quantizer. export.py validates that the group size divides each tensor's total element count. That isn't sufficient. The matmul walks each row with for (j = 0; j <= n - GS; j += GS), so if the group size doesn't divide the row length n, the trailing n % GS values of every row are never read. stories15M has dim = 288, and 288 % 64 = 32 — the default group size silently ignores 11% of every row. My quantizer requires the group size to divide dim and hidden_dim, and auto-reduces 64 → 32.
A portability bug caught by cross-compilation: size_t used without including <stddef.h>, which glibc happens to mask and a freestanding target does not.
Accomplishments that I'm proud of
Bit-exactness, not "within tolerance." Every group accumulates into an int32 before touching float, and integer addition is associative — so the SIMD paths must match the scalar baseline exactly. The harness asserts equality. End-to-end, generation at temperature 0 with a fixed seed produces byte-identical text across kernels, verified with cmp in CI.
A benchmark built to be argued with. Honest baseline, prefill and decode never blended, best-of-N timings with reported spread, hardware capabilities dumped every run so you can confirm i8mm was actually present, and odd-m/odd-d edge cases exercised explicitly because tail handling is where tiling bugs hide.
Reproducibility as a feature. Fork it, press Run workflow, get the table on real Arm64. No hardware required — and scripts/verify_qemu.sh validates correctness with no Arm hardware at all.
What I learned
That the honest ceiling is often the more interesting result. The decode speedup is 1.19x, not 3x, and that's not a tuning failure — matrix-vector streams the entire weight matrix with no reuse, so it's memory-bound. SDOT lifts arithmetic throughput from ~16 to ~20 GOP/s and then bandwidth caps it. The same kernels hit 3.2x the moment there are two activation rows to work with. Understanding why a number is what it is proved more valuable than making it bigger.
I also learned how much correctness infrastructure pays for itself under time pressure. Cross-compiling and emulating cost about thirty minutes and caught three real bugs — two of which would have produced plausible-looking but wrong output.
What's next for NeonForge
Batched prefill in the forward pass. This is the clear gap. The 3.2x i8mm result currently lives in a prefill-shaped microbenchmark, not in the end-to-end time-to-first-token metric, because llama2.c processes prompt tokens one at a time. Restructuring forward() to batch the prompt through wq/wk/wv/wo/w1/w2/w3 as GEMMs would carry that 3.2x directly into TTFT.
Beyond that: SVE2 and bf16 paths (the Neoverse-N2 CI runner advertises both), multi-threaded scaling numbers, SDOT for the activation quantization step, and extending the harness across Graviton, Ampere and Apple Silicon so the runtime dispatch is validated on genuinely different microarchitectures rather than one.
Log in or sign up for Devpost to join the conversation.