Armory

Inspiration

Arm64 architecture is rapidly dominating modern cloud infrastructure and edge computing, yet much of the AI optimization literature and standard deployment pipelines still implicitly assume x86_64 environments.

I set out to explore a fundamental systems engineering question:

How does a real-world Transformer workload behave when optimized and deployed across native Arm64 vs. x86_64 hardware?

I selected DistilBERT for sequence classification (sentiment analysis) to evaluate execution latency, memory footprint, and throughput before and after applying hardware-aware optimization techniques.

The objective was to move beyond theoretical claims and conduct empirical, bare-metal benchmarks on real hardware without emulation.

---

What it does

Armory is an end-to-end NLP inference optimization pipeline and benchmarking harness.

The core pipeline evaluates:

$$\text{DistilBERT (FP32)} \longrightarrow \text{Dynamic INT8 Quantization} \longrightarrow \text{ExecuTorch + XNNPACK} \longrightarrow \text{Native Arm64 Execution}$$

Key Capabilities:

• Multi-Runtime Profiling: Compares standard PyTorch FP32, INT8 Quantized PyTorch, and ExecuTorch runtimes.
• Hardware-Specific Kernels: Targets backend-optimized kernels (QNNPACK for Arm vs. FBGEMM for x86_64).
• Automated Accuracy Verification: Guarantees 100% classification parity between baseline FP32 and quantized representations.
• Empirical Metrics: Measures model size, tail latency, and request throughput under real hardware conditions.

---

The Experiment & Key Findings

The central research question was:

Does model optimization alter the architectural performance dynamics between Arm64 and x86_64?

The empirical data revealed a counterintuitive and compelling result:

Benchmark Data (Real Hardware):

Metric	FP32 / x86_64	FP32 / Arm64	INT8 / x86_64	INT8 / Arm64
Model Size	256.10 MB	256.10 MB	132.29 MB (-48%)	132.29 MB (-48%)
Avg. Latency	65.12 ms	21.07 ms	25.24 ms	25.40 ms
Throughput	15.36 req/s	47.47 req/s	39.62 req/s	39.37 req/s

Insights:

1. FP32 Native Superiority: At FP32 precision, native Arm64 was ~3.1× faster than x86_64.
2. The Quantization Convergence: INT8 quantization provided massive acceleration for x86_64 (reducing latency from 65.12 ms to 25.24 ms — a 2.58× speedup). However, on Arm64, INT8 latency settled around 25.40 ms. As a result, the architectural gap virtually disappeared post-quantization.
3. Memory Footprint: Quantization reduced model size by ~48% (256.10 MB → 132.29 MB), dramatically reducing memory bandwidth pressure.
4. ExecuTorch Acceleration: The ExecuTorch + XNNPACK runtime delivered approximately 1.6× faster inference than the Arm64 FP32 baseline and 2× faster inference than standard INT8.

💡 Core Takeaway: Silicon optimization cannot be predicted purely by theoretical FLOPS. Optimization is an empirical discipline that must be measured on target hardware.

---

How I built it

1. Model Pipeline: Fine-tuned DistilBERT for sequence classification using Hugging Face Transformers and PyTorch.
2. Quantization Engine: Applied PyTorch Dynamic INT8 Quantization (torch.ao.quantization), targeting linear projection layers with architectural engine selection (qnnpack engine for Arm, fbgemm for x86).
3. Edge Runtime Integration: Implemented an experimental export path using ExecuTorch with the XNNPACK delegate, compiling the computational graph into a lightweight binary format.
4. Automated CI/CD Benchmarking: Configured automated GitHub Actions workflows executing across native Arm64 cloud runners to ensure reproducible, isolated testing environments.
5. Validation Suite: Built automated parity harnesses to evaluate cosine similarity and prediction alignment across validation splits.

---

Challenges

• ExecuTorch & PT2E Quantization Constraints: The modern PyTorch 2.x Export (PT2E) quantization path encountered upstream dependency conflicts with specific ExecuTorch delegates. Rather than presenting an unstable pipeline, I scoped the production ExecuTorch export to FP32 with XNNPACK and clearly documented the upstream limitation.
• Cross-Architecture Profiling Consistency: Ensuring fair, deterministic measurements across different instruction sets (AVX-512 vs. Arm Neon) required strict warm-up loops and timing isolation.
• Memory Bandwidth vs. Compute Bound Workloads: Distinguishing between memory-bandwidth-bound and compute-bound bottlenecks during quantization analysis.

---

What I learned

• Architecture-Dependent Scaling: Optimization strategies do not scale uniformly across different CPU architectures. Quantization may completely shift relative hardware advantages.
• Integrity in Engineering: Surprising benchmark results often contain more engineering value than anticipated ones. Documenting performance trade-offs transparently yields better architectural decisions.
• Systems-Level AI Deployment: Profiling memory bandwidth, runtime delegates, and operator kernels is just as critical as model architecture design for real-world deployment.

---

What's next

• End-to-End PT2E Quantization: Integrate stable INT8 quantization within the ExecuTorch pipeline once upstream PyTorch 2.x delegate interfaces mature.
• Broader Silicon Benchmarking: Expand benchmarks to AWS Graviton3/4, Apple Silicon (M-series), and edge compute modules (Raspberry Pi 5, NVIDIA Jetson).
• Kernel-Level Profiling: Conduct low-level instruction profiling using perf and XNNPACK microbenchmarks to inspect SIMD vectorization efficiency.

Built With

• arm
• arm64
• distilbert
• executorch
• githubactions
• huggingface
• machine-learning
• natural-language-processing
• python
• pytorch
• quantization