Voxel - dispatching on microarchitecture, not just ISA

Track 3 · Mobile AI · Pixel 8 (1x Arm Cortex-X3, 4x Arm Cortex-A715, 4x Arm Cortex-A510) + M2 Max

Every inference runtime we examined - ggml, ONNX Runtime, ExecuTorch - dispatches on ISA features: does this chip have i8mm, dotprod, sve2. None reads MIDR. So the same code path runs on an in-order Cortex-A510, an out-of-order A715, and an X3 - in the same die, in the same process, in the same barrier.

We built a from-scratch INT8 engine to find out what that costs. It produces byte-identical output to llama-cli --temp 0 on Mac and Android, and it dispatches per core.

The model is SmolLM2-135M-Instruct, Q8_0, and the small size is deliberate: the product here is an optimized scheduling policy and an on-device agent that turns "set an alarm for 7am" into a fired Android intent. The metric that matters is time to first action, not chat quality - so the whole system is tuned for the sub-second, batch-1, tool-call path that a phone assistant actually runs.

---

1. What we won

Result	voxel	llama.cpp
Time to first action, X3	0.54 s	0.65 s	+17%
Time to first action, A715	0.75 s	1.02 s	+26%
Auto-policy vs llama.cpp's best config	0.60–0.65 s / 2.40 J	0.66–0.67 s / 2.58 J	+7% energy
Capability-weighted prefill, 9 cores	137.3 t/s	our own even split, 69.2	+98%

The last row is the point of the project. The first three are what a user feels.

Read row 4 carefully: it is a self-comparison, the same engine with scheduling on versus off, median of 5 rounds (round-to-round range 76.4-139.1). llama.cpp is far faster than both in absolute 9-core prefill - 1049 t/s - and we say so in section 5. The claim is about the scheduling policy, not about beating ggml at GEMM.

The auto-policy (--policy=auto) reads every core's MIDR from the process's affinity mask at startup and builds two pools: prefill weighted across all nine cores (7:3:1), decode narrowed to the single fastest cluster present. It falls back correctly on homogeneous masks. Both metrics beat llama.cpp's best single configuration.

That comparison is the developer-experience argument too. Getting llama.cpp to its best number on this phone means knowing to pass -t 4, knowing which four cores, and pairing it with a matching taskset mask - folk knowledge, per device. voxel takes one flag with no arguments, --policy=auto, reads the hardware, and beats that hand-tuned configuration. Nothing to measure, nothing to guess, nothing to re-tune on the next SoC.

---

2. The optimization strategy

Prefill and decode are bound by different resources. Almost every result follows from taking that seriously.

Prefill - compute-bound. smmla (2×8 × 8×2 INT8 MAC, one instruction), weights prepacked into tile order at load, 4×4 output tiling. Profiling showed the strided gather was 39% of GEMM time; prepacking removed it. Speedups vs our scalar reference, T=24 SmolLM2 shapes: 2.2–2.5× on X3, 2.7–8.2× on A715, 2.9–5.5× on A510.

Decode - bandwidth-bound. smmla buys nothing at M=1. Different kernel: blockwise Q8 GEMV via sdot, 4 output rows per loaded activation block, one asm! call per row span. Crossing the Rust→asm boundary per 32-element block cost 16.65×; amortizing it was worth more than any instruction choice.

Load We were doing fs::read of 145 MB, then dequantizing every Q8_0 weight to f32 and requantizing straight back to int8. Our block size was already 32 - the same as GGUF's QK8_0. The round trip was recomputing a value the file already contained. mmap + direct block walk: load 1.00 s → 0.30 s. No kernel changed and it made weights byte-identical to GGUF by construction.

Scheduling - the part that's new. ggml splits rows evenly, so the X3 finishes and spins at the barrier waiting for a core 7× slower. That's why the folk advice for llama.cpp on Android is -t 4 - use the big cores, turn off half the silicon. We pin each worker with sched_setaffinity, read its MIDR, and split rows proportionally. On prefill that's +98%. On decode it's worth nothing, because decode saturates DRAM at about four cores - measured, not assumed: contended calibration collapses the 7:3:1 capability ratio toward parity.

---

3. The demo

Live, on-device, text in, tool-call out, a real Android intent fired at the end. Same crate, same model, same kernels as every other benchmark below.

cd voxel && cargo ndk -t arm64-v8a build --release --bin chat
adb push target/aarch64-linux-android/release/chat /data/local/tmp/voxel-cli/chat
adb shell /data/local/tmp/voxel-cli/chat --yolo \
  /data/local/tmp/voxel/models/smollm2-135m-instruct-q8_0.gguf 1

voxel chat -- platform: Android (AlarmClock intent)
(--yolo: tool calls run without confirmation)
type a message (e.g. "set an alarm for 7am"), or /quit to exit.

> set an alarm for 7am
[model tool-call] set an alarm for 07:00 — "Alarm" -- auto-approved (--yolo)
Starting: Intent { act=android.intent.action.SET_ALARM (has extras) }
done: opened the clock app's alarm screen for 07:00

---

4. Two measurements that justify the whole approach

Energy vs throughput, per cluster. Every cluster on the Tensor G3, voxel, 25 s sustained decode, power measured above idle:

Config	t/s	Power above idle	J/token
4× A715 (taskset f0 -t 4)	131.7	0.726 W	0.0054
4× A510 (taskset 0f -t 4)	32.3	0.536 W	0.0143
9 cores (taskset 1ff -t 9)	30.4	1.542 W	0.0383
1× X3 (taskset 100 -t 1)	94.0	2.170 W	0.0507

The surprise is the top and bottom rows: the A715 cluster decodes faster than the X3 prime core while using 9.4× less energy per token. The marquee core is not the right target for LLM decode. It reproduces on llama.cpp too, in the head-to-head tg32 run - 164.7 (A715×4) vs 109.8 (X3) for llama.cpp, 132.7 vs 97.4 for voxel - so it is a property of the silicon, not of our engine.

Prefetch gain by microarchitecture.

	M2 Max	X3	A510
prfm pldl1keep	−8% to −14%	−1% to −6%	+6% to +31%

Not a magnitude difference - a sign inversion, same code, same die. This is the single clearest argument for microarchitectural dispatch, and no ISA-feature check can see it.

Why this generalizes. Nothing here is specific to SmolLM2, to Rust, or to voxel. Every big.LITTLE Arm SoC ships cores with different issue widths and prefetchers behind one ISA feature bitmap, and every major runtime - ggml, ONNX Runtime, ExecuTorch - splits work evenly across them and then waits at a barrier for the slowest. Reading MIDR and weighting the split is ~200 lines against an existing threadpool. It is a change ggml could land, and the phones it would help are already in people's pockets.

---

5. How close we are

Workload	Config	voxel	llama.cpp
Decode	X3 ×1	97.4	109.8	−11%
Decode	A715 ×4	132.7	164.7	−19%
Decode	A510 ×4	32.5	29.2 ± 17.5	parity (inside noise)
Decode	A510 ×1	14.1	21.9	−36%
Prefill	X3, pp24	81.4	599.0	−7.4×
GEMM kernel	X3, single-core	-	-	ggml 7.6–8.3×

Within ~11% of a tuned runtime on single-core decode, from scratch. Prefill is where we lose badly.

We know why the A510 gap is worst. The A510 is in-order - there is no OoO window to hide sdot latency, so ILP must be handed to the core by the code. Our GEMV accumulates into one serial dependency chain per row; ggml uses four independent chains. That single fact retro-explains prepacking helping A510 twice as much as X3, per-call overhead costing A510 most across three separate measurements, and prefetch helping in isolation but regressing under contention.

---

6. What didn't work

• Weight interleaving (ggml's block_q8_0x4 layout): net loss on M2 Max (0.76–0.80×) and X3 (0.93–0.95×). Not shipped.
• Per-core prefetch dispatch, end-to-end: 85.0 t/s vs 88.3 uniform-off and 90.4 uniform-on. The kernel-level sign flip is real; it does not yet convert into an end-to-end win. Under 4-thread A510 contention prefetch costs 4.5%.
• Capability weighting on decode: 69.3 → 53.9 → 40.8 t/s as weighting sharpens. Monotonically worse, because no scheduling policy conjures bandwidth.
• GPU offload at this scale: Metal -ngl 99 277 vs 423 t/s CPU (1.5× slower); CoreML EP 357 ms vs 92 ms CPU (3.9× slower). At sub-200M with batch=1, dispatch overhead exceeds kernel time.
• Per-tensor quantization: faster, and diverged from llama.cpp at token 3. Retracted along with the headline it had produced.
• 9-core decode energy: our own gate rejected it at 5.13 W. Logged as failed, not quietly dropped.

7. Three ways Android silently lies to your benchmark harness

Each found by a number that failed a sanity check. Each corrupts results by 2–70×.

1. hardware_concurrency() ignores taskset. Pinned to 4 cores, llama-cli spawns 9 threads: 0.2 t/s vs 14.0 t/s with -t 4 - a 70× error. Present in llama-cli, llama-bench, and test-backend-ops, which has no flag to override it. We hit it three times, including once in our own harness.
2. Doze parks the governors. Screen off and idle, cpu8 sits at 500 MHz of 2914. 9-core decode reads 1.5 t/s against a ~69 t/s baseline.
3. Screen-off throttles sustained load, unevenly. Same workload, display off: voxel X3 drops 6× (97 → 16 t/s), llama-cli only 2.8×. Every energy figure we published before finding this was measured at an uncontrolled operating point - which is exactly why they disagreed by 13×.

Without root you cannot force a governor, so all three must be controlled procedurally. make bench does.

---

8. Reproducing

make bench            # host: tests + kernel sweep + profile
make bench-android    # per-cluster, pinned, both engines

Every kernel is gated on every run against a scalar reference with randomized inputs - 20/20 passing, including tail rows, oversized prefetch distances, and bit-identity between the pinned multi-threaded pool and single-threaded execution.

Protocol, on every table: unplugged, wireless ADB, device held awake, explicit -t matching taskset popcount, 3 s cooldown, median of 5, power gate 0.1–4 W.

Built With

• arm
• kernels
• python
• rust