ARM Mixquant

Sensitivity-aware mixed precision for ARM. Find the fragile tensors, spend bits there, and keep everything else small.

Comment

Inspiration

Uniform 4-bit quantization is the default almost everyone ships. It is convenient, and it is blunt. Every tensor gets the same squeeze, even though layers do not fail the same way. Some are fragile and tiny. Some are fragile and expensive. Most of the rest can stay small.

The Arm AI Optimization Challenge asked for real optimization, not another local chatbot. That pointed us at a simple question: if we measure which tensors actually break, can we beat the presets people already use, on Apple Silicon, without inventing a new codec?

What it does

arm-mixquant is a sensitivity-aware mixed-precision recipe on top of llama.cpp.

  1. It profiles a Hugging Face model and scores each weight tensor by how much error 4-bit round-to-nearest would introduce, weighted by activations.
  2. It builds a mixed GGUF: a strong low-bit base, then extra bits only on the fragile families.
  3. It benches that file against stock llama.cpp quants on the same Arm Mac, with the same prompts.

On Qwen2.5-1.5B-Instruct the default recipe is IQ4_XS everywhere, Q8_0 on every GQA attn_k / attn_v, and Q8_0 on only the last two ffn_down. The result is 878 MiB at WikiText perplexity 8.92 (lower is better), versus stock Q4_K_M at 940 MiB and 8.97. It is also smaller and better than IQ4_NL, and better than IQ4_XS at matched size.

This is not a chat product. python3 -m bench.demo_prompt runs one prompt on mixed versus stock so you can see the model actually answer on this machine.

Perplexity is the usual quality score:

$$ \mathrm{PPL} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log p(x_i \mid x_{<i})\right) $$

Lower $\mathrm{PPL}$ means the quantized model is less surprised by the eval text.

How we built it

Mac first, because that is the Arm device we had.

  • Profiler: PyTorch hooks, activation-weighted 4-bit RTN error, written to results/sensitivity.json.
  • Recipe search: family-aware upgrades (all cheap KV, top-$k$ expensive FFN) under a size band around the stock baseline.
  • Export: llama-imatrix plus llama-quantize --tensor-type, Metal build of llama.cpp.
  • Harness: size, TTFT, tokens/sec, RSS, perplexity, and KL versus F16, then Pareto and layer charts.

The same GGUF is what you would copy to Graviton. iPhone is the same idea exported to ExecuTorch .pte, not a drop-in Mac file. Two commands reproduce the Mac path: scripts/bootstrap_mac.sh and scripts/phase1_mac.sh.

Challenges we ran into

Llama 3.2 was gated without a Hugging Face token, so we switched the primary model to open Qwen2.5-1.5B-Instruct instead of blocking the whole project.

Beating Q4_0 was easy. Beating IQ4_XS at matched size was not. Early mixes were slightly better on perplexity but too big. The fix was the finding we now lead with: GQA KV tensors are fragile and tiny, so protecting all of them is almost free, while late FFN is the only expensive upgrade worth buying.

We also tried a QuaRot-style rotation (fused Hadamard on $V$/$O$). It was numerically equivalent in FP32 and still did not beat the unrotated recipe, so the file we ship is unrotated. Honesty was harder than adding another flag.

CUDA-only methods (GPTQ, AWQ, QuIP#) would not install cleanly on this Mac. We treated that as out of scope and compared against the llama.cpp presets you can actually run on Arm.

Accomplishments that we're proud of

One mixed GGUF that is 63 MB smaller than Q4_K_M with better perplexity, and that also wins against IQ4_NL and matched-size IQ4_XS.

A finding you can see in ten seconds on the heatmap: protect all cheap KV, 8-bit only the last FFN layers.

A repo other people can rerun: Apache-2.0, two scripts, charts in results/, no weights in git.

What we learned

Sensitivity is not "make everything 8-bit." The interesting budget is which tensors. On this GQA model, KV is the cheap win and late ffn_down is the expensive one. Uniform presets waste bits on early FFN and under-protect KV.

We also learned that a rotation paper does not automatically become a better GGUF. If the number does not move, do not claim it.

Lower perplexity is better. A single generated sentence is not a benchmark. The live demo is only proof that the file runs.

What's next for ARM Mixquant

Apply the same ranker to other open models via PRIMARY_ID.

Land the same protect list on-device as ExecuTorch .pte and publish iPhone numbers, not only Mac ones.

Try a K-quant-only mix (for example Q6_K on KV) if mixed IQ+Q8 ever pays a kernel-mix tax on a given machine.

Keep the rule: bits go where the model breaks, not where the preset says.

Built With

Share this project:

Updates