ARMScale

Inspiration

ARMScale was inspired by a simple problem: AI inference performance depends heavily on how a model is configured, but choosing the best configuration is often based on guesswork. We wanted to build a system that could measure different configurations and automatically identify the best trade-offs between speed, throughput, and model size.

Our goal was to make optimization evidence-based rather than assumption-based, especially for resource-constrained and ARM-oriented deployment environments.

What We Built

ARMScale is an autonomous inference optimization platform that benchmarks and evaluates combinations of:

3 model quantizations: Q4_K_M, Q5_K_M, Q8_0
4 thread configurations: 2, 4, 6, 8
3 context sizes: 1024, 2048, 4096

This creates a complete search space of:

$$3 \times 4 \times 3 = 36$$

configurations per workload.

ARMScale measures latency, throughput, load time, and model size, then uses multi-objective scoring and a 3D Pareto analysis across:

$$\text{Latency} \downarrow,\quad \text{Throughput} \uparrow,\quad \text{Model Size} \downarrow$$

The system can generate recommendations for four objectives: speed, throughput, size, and balanced performance.

How We Built It

The backend was built in Python with a modular optimization engine, benchmark runner, configuration generator, scoring system, experiment registry, recommender, and REST API.

Inference benchmarking uses llama-cpp-python and GGUF models from the Qwen2.5 0.5B Instruct model family.

We implemented automated testing for configuration generation, validation, scoring, Pareto optimization, API endpoints, experiment discovery, and recommendation logic.

The frontend provides a dashboard for exploring benchmark results, comparing configurations, and understanding the optimization results visually.

Results

During Phase H, ARMScale evaluated all 36 configurations for two workloads:

short_generation
context_stress

This resulted in 72 configurations and 504 authentic inference executions, with warmup runs excluded from statistical measurements.

For short generation, the best configuration was:

Q4_K_M + 6 threads + 4096 context

with 1801.65 ms mean latency and 48.14 tok/s.

For context stress, the best speed/throughput configuration was:

Q8_0 + 8 threads + 2048 context

with 4310.92 ms mean latency and 25.82 tok/s.

The system also identifies size-efficient Pareto-optimal configurations rather than simply selecting the fastest model.

What We Learned

One of the biggest lessons was that more threads, larger context windows, or higher-precision quantization do not automatically produce better performance. The optimal configuration depends on the workload and the objective being optimized.

We also learned the importance of reproducible benchmarking, cryptographic model verification, automated testing, and clearly separating development hardware measurements from target-hardware claims.

Challenges

The main challenges were designing a reliable multi-dimensional search space, preventing invalid or duplicate configurations, collecting consistent benchmark measurements, handling different optimization objectives, and correctly calculating Pareto dominance across multiple dimensions.

Another important challenge was avoiding misleading hardware claims. Phase H was benchmarked on a local x86_64/AMD64 development machine, so these results are explicitly not presented as Google Axion or Arm64 performance measurements.

What's Next

ARMScale provides the optimization foundation for future deployment on ARM64 hardware. The next step is to run the same benchmark methodology on real ARM64 infrastructure and compare the resulting optimization landscape against the x86_64 development results.