-
-
Real, Arm-verified benchmark: 47% smaller model size after quantization, measured live on genuine aarch64 hardware via CI.
Inspiration
Most AI models are too large and too slow to actually run well on real, constrained hardware — and Arm is quickly becoming the dominant architecture for exactly that kind of efficient, low-power computing, from mobile devices to cloud servers like AWS Graviton and Oracle Ampere. We wanted to build something that didn't just claim to be optimized for Arm, but actually proved it, with real, reproducible numbers anyone could verify themselves.
What it does
ArmTune is a command-line tool that takes any Hugging Face model — a chatbot, a speech-to-text model, or a vision classifier — and automatically:
- Loads the model
- Measures its baseline size and speed
- Quantizes it to INT8 using the correct backend for the CPU it's running on
- Measures it again after quantization
- Generates a clean HTML report showing the real before/after difference
One command, any model, real numbers.
How we built it
The core is PyTorch's dynamic quantization, wrapped in a small Python CLI. We built a custom benchmarking harness that times model size, latency, and tokens/sec, and a report generator that turns those numbers into a readable HTML page.
To make sure our results were genuinely trustworthy — not just numbers from our own laptop — we set up a GitHub Actions workflow that runs the exact same benchmark automatically on a real Arm CPU (a Cobalt 100-based aarch64 machine, free for public repos). That means anyone, including judges, can open our repo's Actions tab and re-run the benchmark themselves to verify our results.
Challenges we ran into
Our first real run on Arm hardware crashed immediately with RuntimeError: unknown architecture. It turned out PyTorch's default quantization backend, fbgemm, only supports x86 (Intel/AMD) CPUs — Arm requires a different backend called qnnpack. We fixed this by having ArmTune automatically detect the CPU architecture at runtime and select the correct backend.
After fixing that, our real benchmark revealed something more interesting than a clean win: quantization reduced model size by 47%, but increased latency by 2.2x for this particular model and workload. Rather than hide that or cherry-pick a different result, we investigated why: dynamic quantization re-quantizes activations on every forward call, and for small-batch, token-by-token text generation, that per-step overhead outweighs the compute savings. It's a real, explainable trade-off, not a bug — and we documented it honestly in our results.
Accomplishments that we're proud of
- A real, reproducible 47% model size reduction, independently verifiable on genuine Arm hardware via public CI logs
- Catching and correctly fixing a real Arm-specific compatibility bug (fbgemm vs qnnpack), not just following a tutorial
- Choosing to report an honest, mixed result with a clear technical explanation instead of manufacturing a cleaner-looking claim
What we learned
Optimization on Arm isn't a single "faster/smaller, always" story — different quantization strategies have real, workload-dependent trade-offs. Dynamic quantization favors memory savings over latency for small-batch autoregressive generation; static quantization or larger batch sizes would likely favor speed instead. Reproducibility also matters as much as the result itself — a benchmark judges can independently re-run is worth more than a screenshot.
What's next for ArmTune - AI Model Optimizer for Arm CPUs
- Add a static quantization path optimized for latency-sensitive use cases
- Add real accuracy/quality checks (word error rate for speech models, perplexity for language models) to fully cover model-quality impact, not just size and speed
- Add ExecuTorch export for true on-device mobile benchmarking, extending this from cloud/server Arm CPUs to mobile Arm chips
Log in or sign up for Devpost to join the conversation.