ArmBench CI

Native Arm64 CI that quantizes an ONNX intent classifier to INT8 and publishes reproducible accuracy, size, latency, throughput, and memory trade-offs.

Comment

Challenge Track

Cloud AI

Inspiration

AI optimization claims are often published without evidence from the target hardware—or without making the accuracy trade-off visible. ArmBench CI turns that gap into an auditable, repeatable workflow: optimize a model, run it on native Arm64 infrastructure, and publish both performance and quality evidence.

What it does

ArmBench CI trains a deterministic compact intent classifier, exports it to ONNX, applies ONNX Runtime dynamic INT8 weight quantization, and benchmarks FP32 and INT8 on a native Arm64 GitHub Actions runner. It publishes machine-readable and human-readable evidence for model size, accuracy, p50/p95 latency, repeated full-test-set throughput, peak resident memory, architecture, and package versions. A policy gate fails CI if the run is not native Arm64, if the declared sequential single-thread protocol or high-water RSS evidence is missing, or if the optimization misses its size, throughput, latency, or accuracy limits. The gate recomputes every derived metric from finite raw measurements.

How we built it

The benchmark uses Python, scikit-learn, skl2onnx, ONNX Runtime, psutil, and GitHub Actions. A seeded synthetic multiclass intent-style workload trains an MLPClassifier; skl2onnx exports the FP32 graph, and ONNX Runtime dynamically quantizes its weights to QInt8. GitHub's ubuntu-24.04-arm runner executes the same benchmark and validator, then publishes JSON, Markdown, and a concise gate decision as workflow artifacts.

Measurement protocol

Latency is measured after warm-up across five independent trials. Throughput is reported as the median of nine full-test-set trials. Accuracy is evaluated on a held-out test split. Both variants use the same sequential, single-threaded ONNX Runtime policy, and memory is the process high-water RSS rather than a post-run point sample. This repeated protocol reduces the influence of runner noise and makes the headline numbers reproducible from CI.

Verified native Arm64 result

The policy-enforced native run produced:

  • 73.39% smaller model
  • 1.06× p50 latency speedup
  • 1.12× p95 latency speedup
  • 2.86× median throughput speedup
  • -0.0075 accuracy delta

The result demonstrates a practical cloud-AI trade-off: substantially lower storage and transfer cost, materially higher batch throughput, and only a small measured accuracy change. It does not claim that INT8 accelerates every model.

Challenges

Shared CI runners are noisy, especially for microbenchmarks. A single timing pass can overstate or understate improvement. We addressed this with repeated trials, medians, warm-ups, fixed seeds, controlled threading, explicit architecture checks, committed reports, and machine-readable policy thresholds.

Accomplishments

ArmBench CI is not a screenshot-only demo. The benchmark, optimization, native Arm64 workflow, raw evidence, thresholds, and corrected demo are public and rerunnable. Every claim can be traced to code and a successful native CI execution. Fifteen adversarial tests cover policy tampering, missing metrics, protocol drift, invalid RSS, submission-copy consistency, and the valid reference report.

What we learned

Optimization should be treated as a multi-objective engineering decision. Size, tail latency, throughput, accuracy, reproducibility, and provenance all matter; no single metric is enough.

What's next

Next steps include larger Arm-relevant ONNX models, operator-level profiling, energy and cost-per-inference measurements, and regression tracking across ONNX Runtime and Arm runner updates.

Evidence

Built With

Share this project:

Updates