ArmDX

dashboard
results achieved
Results
model used
optimizations

Inspiration

Optimizing LLM inference on Arm64 hardware involves many variables, including model quantization, CPU kernels, and runtime configurations. Developers can spend a lot of time manually testing combinations or relying on assumptions about which configuration will perform best.

We wanted to replace that guesswork with measured evidence.

That led us to build ArmDX, a console that lets a user define an optimization goal and automatically benchmark relevant configurations directly on target Arm64 hardware.

What it does

ArmDX provides three optimization goals:

Make it faster — find a faster runtime while keeping the model at Q8_0.
Keep quality — maintain the high-precision Q8_0 model while finding the best runtime configuration.
Reduce disk size — evaluate quantized models and select a smaller model configuration.

The user starts a Microsoft Azure Cobalt 100 Arm64 VM, establishes an SSH tunnel, and opens the ArmDX local web dashboard.

From the dashboard, the user selects an optimization goal. The request is sent to the Arm64 environment, where ArmDX executes reproducible llama-bench runs across predefined llama.cpp configurations. The measured results are then returned to the dashboard, where ArmDX selects and presents the best configuration for the chosen goal.

How we built it

ArmDX uses a local HTML/CSS/JavaScript dashboard connected to a FastAPI orchestration service running on an Azure Cobalt 100 Arm64 VM.

The backend orchestrates llama-bench across three runtime configurations: stock CPU, Arm KleidiAI, and KleidiAI + OpenBLAS. It also evaluates different Qwen2.5 1.5B Instruct GGUF quantizations for the disk-size optimization.

The system preserves the raw benchmark output as JSON evidence for each job, allowing the displayed results to be traced back to actual benchmark executions.

The complete workflow is:

Local dashboard → SSH tunnel → Azure Cobalt 100 Arm64 → benchmark execution → raw evidence → results returned to dashboard

Challenges we ran into

One of our biggest challenges was making a remote Arm64 benchmarking workflow feel simple from the user's perspective.

We had to connect a local interface to the remote Arm environment without exposing the backend publicly, while still allowing benchmark jobs to be triggered and results to be returned reliably.

Another challenge was making the optimization evidence trustworthy. We did not want the interface to simply display predetermined performance numbers, so we designed the system around actual llama-bench executions and preserved the raw JSON output.

We also had to make comparisons fair by keeping the workload and benchmark conditions consistent across candidate configurations.

Accomplishments that we're proud of

We successfully ran ArmDX on a live Microsoft Azure Cobalt 100 Arm64 VM and measured meaningful performance differences between Arm optimization configurations.

For Make it faster, ArmDX selected Q8_0 + KleidiAI and improved prompt processing from 73.95 tok/s to 97.28 tok/s, a 31.5% improvement.

Token generation improved from 37.51 tok/s to 42.68 tok/s, a 13.8% improvement.

For Keep quality, the selected Q8_0 + KleidiAI + OpenBLAS configuration reached 97.41 tok/s prompt processing, a 31.1% improvement over the stock baseline.

For Reduce disk size, ArmDX selected Q4_0 and reduced the model size from 1570 MB to 892 MB, approximately 43.2% smaller.

We are especially proud that these numbers come from measured executions on real Arm64 infrastructure rather than simulated or hard-coded results.

What we learned

We learned that Arm optimization is highly dependent on the workload and the objective being optimized.

The configuration that maximizes inference speed is not necessarily the one that minimizes model size, and a smaller quantized model can involve different performance trade-offs from a higher-precision model.

We also gained practical experience with Arm64 cloud infrastructure, llama.cpp, KleidiAI, quantization, remote orchestration, SSH tunneling, reproducible benchmarking, and preserving raw performance evidence.

Most importantly, we learned that optimization becomes much more useful when it is treated as an empirical measurement problem rather than a guessing problem.

What's next for ArmDX

The next step is to expand ArmDX beyond the current predefined benchmark candidates.

We want to support more models, more Arm runtime configurations, and additional optimization objectives such as memory efficiency and cost-performance trade-offs.

We also want to make the benchmarking workflow easier to deploy across different Arm64 cloud environments so that developers can compare and reproduce results on their own infrastructure.

The long-term goal is to make ArmDX a practical optimization layer for developers who want to get the best measurable performance from LLM inference on Arm.