Inspiration
AI performance is often discussed in terms of model size and hardware specifications, but that does not answer a more useful engineering question: where is the workload actually spending its time, and what changes when we optimize it?
We wanted to build a practical laboratory for investigating that question on Arm architecture.
Instead of starting with a large AI model and treating the runtime as a black box, we started from the fundamentals: CPU architecture, AArch64, registers, SIMD, NEON, cache locality, and memory bandwidth. We then built controlled experiments before moving toward AI workloads.
This led to Arm AI Benchmark Lab — an interactive performance-engineering environment for measuring, comparing, and understanding AI workloads across baseline and optimized configurations.
What it does
Arm AI Benchmark Lab provides a unified environment for AI performance analysis.
It currently supports three representative workload categories:
- Llama 3 8B — Large Language Model
- Whisper Large v3 — Speech Recognition
- Vision Transformer / ViT Base — Computer Vision
For each workload, the laboratory compares baseline and optimized configurations using metrics such as:
- Latency
- Throughput
- Memory footprint
- Model size
- Percentage improvement
The project also contains controlled CPU experiments that demonstrate the architectural principles behind AI optimization.
For example, our SIMD experiment processes 10 million FP32 elements and compares scalar execution against vectorized execution. The recorded experiment measured 0.8028 seconds for the scalar implementation versus 0.0172 seconds for the vectorized implementation, a 46.68× speedup.
A second experiment investigates memory locality using a 64 MB buffer and compares sequential access with a stride of 64. The experiment recorded an 11.67× per-element access penalty for the strided pattern.
The project exposes these capabilities through both an interactive CLI and a React-based web dashboard.
How we built it
The project was developed as a five-phase engineering workflow:
Phase 1 — Arm Architecture Fundamentals
We documented Arm vs ARM64 vs AArch64, AArch64 registers, ISA concepts, SIMD/NEON, cache hierarchy, memory bandwidth, and Arm Neoverse infrastructure.
Phase 2 — Controlled Micro-Experiments
Before introducing AI frameworks, we created reproducible experiments for:
- Scalar vs vectorized computation
- Cache and memory-stride locality
Each experiment follows:
Question → Hypothesis → Implementation → Benchmark → Result → Explanation
Phase 3 — Benchmarking Engine
The Python backend uses PyTorch, NumPy and system telemetry to execute CPU tensor workloads and collect latency, throughput and memory measurements.
The benchmark engine also calculates percentage changes between baseline and optimized configurations.
Phase 4 — AI Optimization Matrix
The workload definitions model optimization strategies including FP16, INT8/INT4 quantization, vectorized execution, Arm NEON, KleidiAI-oriented acceleration, and optimized inference runtimes such as ONNX Runtime and llama.cpp.
Phase 5 — Interactive Product
We built a full-stack interface around the benchmarking engine:
- React 18 + Vite frontend
- FastAPI backend
- Interactive workload selection
- Hardware information panel
- Metric comparison cards
- Performance comparison tables
- Telemetry visualizations
- Interactive CLI benchmark runner
The repository also includes separate Arm64 and AMD64 Dockerfiles and a deployment script for Arm-based cloud environments.
Challenges we ran into
1. Separating architecture concepts from actual performance
It is easy to say that SIMD, cache locality, or quantization improves performance. It is much harder to demonstrate why.
We therefore built small controlled experiments first. This allowed us to isolate individual effects before introducing the complexity of AI inference.
2. Benchmarking different workload types consistently
An LLM, speech-recognition model, and vision model have very different computational characteristics.
We had to establish a common reporting structure around latency, throughput, memory, model footprint, and percentage changes while still preserving workload-specific behavior.
3. Supporting both local development and Arm64 deployment
Development was performed on a Windows/x86 environment, while the ultimate target is Arm64.
To address this, the project includes separate container configurations for linux/amd64 and linux/arm64, along with deployment tooling intended for Arm-based cloud instances.
4. Making the benchmark interactive
A benchmark should not require manually reading terminal output or modifying source code for every test.
We therefore built both a CLI interface and a web dashboard backed by a FastAPI API so that workloads and benchmark parameters can be explored interactively.
5. Keeping performance measurements explainable
A single performance number is not enough.
The project therefore reports the baseline, optimized configuration, measured metrics, and calculated improvement together so that users can understand what changed and why.
Accomplishments that we're proud of
- Built a complete Arm-focused AI benchmarking laboratory rather than a single benchmark script.
- Created controlled SIMD and cache-locality experiments before introducing AI workloads.
- Recorded a 46.68× speedup in the scalar-versus-vector experiment.
- Demonstrated an 11.67× memory-access penalty from poor stride locality.
- Added workload definitions for LLM, speech, and computer-vision inference.
- Built a Python benchmarking engine with CPU tensor execution and system telemetry.
- Built a FastAPI REST backend for benchmark execution and system information.
- Built an interactive React/Vite dashboard for visual comparison of benchmark results.
- Added CLI support for running individual workloads and the complete experiment suite.
- Added Arm64 Docker support alongside AMD64 containers.
- Added cloud deployment tooling targeting Arm-based infrastructure.
- Structured the entire project as a learning-to-engineering pipeline rather than hiding the optimization process behind a black box.
What we learned
The biggest lesson was that AI optimization starts below the model level.
Before optimizing an LLM, it is important to understand what the CPU is doing with data. Registers, vector instructions, cache lines, memory bandwidth, thread utilization, numerical precision, and runtime implementations can all influence the final result.
We also learned that benchmarking needs experimental discipline. A performance number without a baseline, workload definition, environment, and explanation is difficult to interpret.
The project reinforced the importance of moving through the stack systematically:
Architecture → Microbenchmark → Runtime → AI Workload → Optimization → Deployment
We also learned that Arm64 is not simply a different CPU target. It requires thinking about the complete software stack, including architecture-specific instructions, optimized libraries, inference runtimes, containers, and cloud infrastructure.
Most importantly, we learned to treat performance engineering as an empirical process: measure first, form hypotheses, optimize, and measure again.
What's next for Arm AI Benchmark Lab
The current project establishes the laboratory foundation. The next stage is to move from a benchmark framework toward deeper native Arm performance analysis.
Planned improvements include:
- Run the complete benchmark suite on native Arm64 hardware.
- Compare x86-64 and Arm64 results under controlled conditions.
- Add real model inference using downloadable model weights.
- Expand experiments for NEON and SVE/SVE2.
- Integrate deeper Arm performance profiling.
- Add more quantization configurations such as INT8 and INT4.
- Benchmark additional inference runtimes and Arm-optimized libraries.
- Track CPU utilization, cache behavior, memory bandwidth, and power/efficiency metrics.
- Store benchmark runs so results can be compared across machines and configurations.
- Expand the dashboard into a complete performance-analysis laboratory.
- Build reproducible Arm64 cloud benchmark pipelines using Arm-based instances.
The long-term goal is to turn Arm AI Benchmark Lab into a reproducible open-source environment where developers can answer a practical question:
Given an AI workload, what is limiting its performance on Arm, and which optimization actually makes a measurable difference?
Log in or sign up for Devpost to join the conversation.