Optimization claims with receipts
ArmForge CI is a reusable Cloud AI workflow that turns a pinned FP32 ONNX model into a per-channel signed INT8 build, benchmarks both on native Arm64, validates output quality, and publishes auditable evidence. It addresses a practical developer problem: optimization claims often lack a pinned baseline, architecture proof, fair controls, or a reusable way to repeat the result.
Judge quick review: baseline → technical change → Arm64 proof
Area FP32 baseline Technical change Native Arm64 result
Model Pinned sentence-transformers/all-MiniLM-L6-v2 ONNX FP32 model Per-channel signed INT8 dynamic quantization 86.22 MB → 21.94 MB (74.55% smaller)
Inference ONNX Runtime CPU execution Same provider, graph settings, threads, deterministic inputs, warmups, and measured runs; only the quantized model changes 19.31 ms → 7.83 ms mean latency (59.47% lower); 51.78 → 127.76 items/s (146.74% higher)
Memory Isolated FP32 worker Independent INT8 worker prevents allocator/memory-map carryover 135.30 MB → 56.04 MB RSS delta (58.58% lower)
Quality FP32 output tensors Compare every output tensor after quantization cosine similarity 0.98344814, above the enforced 0.98 threshold; all outputs finite
These are from a successful public ubuntu-24.04-arm GitHub-hosted runner run, not emulation or cross-compilation. The repository links the raw JSON, Markdown report, workflow logs, artifact, dashboard, model revision, SHA-256, and full methodology.
What I built
The Python package uses ONNX Runtime and its quantization APIs. It downloads the default MiniLM ONNX model at an exact Hugging Face revision and verifies its SHA-256 before processing. The benchmark controller launches FP32 and INT8 inference in separate worker processes. This is deliberate: it prevents one model's allocator, memory maps, and runtime state contaminating the other's RSS evidence.
Both variants receive the identical deterministic workload and share runtime version, CPU execution provider, graph optimization level, thread count, batch size, sequence length, warmups, and measured runs. The gate fails if the runner is not native Arm64, if any output is non-finite, or if output cosine similarity is below 0.98. It reports speed transparently rather than making it a brittle pass/fail threshold on shared CI.
Every run publishes mean, P50, P95, min/max latency, throughput, model size, process RSS increase, cosine similarity, numerical error, architecture, OS, Python, ONNX Runtime, provider, thread settings, raw JSON, Markdown, workflow summary, downloadable artifact, and a public dashboard.
Why this matters for Arm developers
Public GitHub repositories can use native Arm64 runners without buying hardware or opening a paid cloud account. ArmForge CI turns that access into a repeatable decision gate: model owners can verify native execution, produce an architecture-appropriate INT8 build, quantify size/latency/throughput/memory changes, prevent unacceptable drift, and attach inspectable evidence to a pull request or release.
It is reusable beyond this project. The composite GitHub Action accepts a repository-owned ONNX model plus workload and quality settings, then writes the same JSON and Markdown evidence files for a project to archive or enforce in CI.
Reproduce it
On Arm64 Linux: python -m pip install -e ".[dev]" armforge --runs 75 --warmups 15 --threads 4 --batch-size 1 \ --sequence-length 128 --output benchmarks --require-arm64 pytest
The published run used 75 measurements after 15 warmups, four ONNX Runtime intra-op threads, batch size 1, and sequence length 128. CI performance has host variance, so reruns are directionally comparable rather than dedicated-lab measurements.
What I learned
Optimization tooling needs provenance and quality gates as much as it needs speed. A headline improvement is not trustworthy when a model revision moved, the target architecture is ambiguous, output quality silently regressed, or memory measurements share process state.
What's next
Static calibration for models that benefit from QDQ quantization, multi-batch matrices, confidence intervals across repeated Arm64 runs, PR comments, model-specific acceptance profiles, and additional Arm inference runtimes.
Apache-2.0 licensed; the default Apache-2.0 model is downloaded from its pinned upstream revision and is not redistributed. ArmForge CI is an independent hackathon submission.
Built With
- arm64
- css
- github-actions
- html
- javascript
- numpy
- onnx
- onnx-runtime
- psutil
- python