ERNIE Inference Optimization on Arm

Accelerating ERNIE inference on AWS Graviton3 with ONNX Runtime, hardware-aware tuning, and INT8 quantization.

Comment

Inspiration

Transformer models are powerful, but deploying them efficiently on CPUs can be challenging because of their computational and memory requirements. I wanted to explore how much inference performance could be improved by combining model/runtime optimization with hardware-aware tuning for Arm server CPUs.

The goal was not simply to apply a list of optimizations, but to benchmark each stage on real Arm hardware and measure the trade-offs between latency, throughput, model size, and prediction quality.

What it does

The project builds an end-to-end inference optimization pipeline for ERNIE 3.0, fine-tuned for Chinese sentiment classification using ChnSentiCorp.

The pipeline starts with a PyTorch FP32 model and progressively applies:

  • PyTorch-to-ONNX export
  • ONNX Runtime inference
  • ONNX graph optimization
  • CPU thread tuning
  • Dynamic INT8 quantization
  • Model quality evaluation
  • Latency, throughput, and model-size benchmarking

The final pipeline was benchmarked on an AWS Graviton3 c7g.xlarge Arm64 instance.

The optimized INT8 model achieved:

  • 3.86x faster inference
  • 3.86x higher throughput
  • 74.81% smaller model size (3.97x reduction)
  • 21.31 ms average latency, compared with 82.34 ms for the PyTorch FP32 baseline
  • 89.17% accuracy, compared with 91.17% before INT8 quantization

All performance results were measured at batch size 1 and sequence length 64.

How we built it

I first fine-tuned nghuyong/ernie-3.0-base-zh for binary Chinese sentiment classification.

I then established a PyTorch FP32 inference baseline before exporting the model to ONNX. ONNX Runtime was used as the CPU inference runtime, after which I benchmarked different graph optimization levels and CPU thread configurations.

The FP32 ONNX model was then dynamically quantized to INT8. I repeated runtime optimization and benchmarking on the quantized model and evaluated both FP32 and INT8 models on the same development set to measure the quality trade-off introduced by quantization.

The final benchmarks were executed on AWS Graviton3 using ONNX Runtime's CPUExecutionProvider.

Challenges we ran into

One of the main challenges was discovering that an optimization does not automatically translate into better real-world performance.

ONNX graph optimization and CPU thread tuning produced little additional improvement for FP32 inference on the target hardware. Dynamic INT8 quantization, however, produced a substantial reduction in latency and model size.

I also encountered practical challenges around ONNX export, dynamic quantization, model artifacts, reproducible benchmarking, and maintaining consistent evaluation between FP32 and INT8 models.

Rather than hiding unsuccessful optimization attempts, I kept them in the results because they demonstrate why inference optimization should be driven by measurement on the target hardware.

What we learned

The biggest lesson was that inference optimization is an empirical process.

Performance depends on the interaction between the model, inference runtime, numerical representation, workload, CPU architecture, and runtime configuration. An optimization that sounds beneficial in theory may have little effect—or even regress performance—on a particular deployment target.

I also gained practical experience with ONNX model export, ONNX Runtime, graph optimization, CPU threading, INT8 quantization, profiling, benchmarking, and deploying transformer inference on Arm64 infrastructure.

What's next for ERNIE Inference Optimization on Arm

The project could be extended by benchmarking additional sequence lengths and batch sizes, evaluating other Arm platforms, and exploring further Arm-specific inference optimizations.

Built With

Share this project:

Updates

Submission history