Inspiration
The Arm ecosystem spans from edge devices to high-performance servers, each with unique hardware capabilities and constraints. We recognized that optimizing machine learning models for this diverse range of Arm architectures requires deep domain expertise in both ML optimization and hardware profiling. The challenge isn't just making models faster—it's making them intelligently adapt to the specific hardware they'll run on. This inspired us to build an automated system that combines hardware-aware optimization with AI-driven search to unlock the full potential of Arm processors.
What it does
Arm-Autopilot is an automated optimization framework that intelligently tailors machine learning models to specific Arm hardware targets. The system:
- Profiles hardware capabilities automatically using Arm Performix, collecting detailed metrics on CPU architecture, memory hierarchy, and performance characteristics
- Defines a configuration space of model optimization parameters (quantization levels, compiler flags, inference engines) that are hardware-aware and vendor-optimized
- Searches the optimization space using Bayesian optimization with a Gaussian Process surrogate model to efficiently explore millions of possible configurations
- Generates Pareto-optimal configurations that balance model accuracy, latency, and memory usage according to deployment constraints
- Exports optimized models to multiple inference frameworks (llama.cpp, ONNX, TensorFlow Lite) ready for production deployment
- Generates detailed reports with performance metrics, deployment profiles, and actionable optimization recommendations
The result is a fully automated pipeline that transforms a generic model into a hardware-optimized, production-ready solution without manual engineering.
How we built it
Arm-Autopilot was architected as a modular Python framework with clear separation of concerns:
- Hardware Profiler (
hw_profiler/) - Integrates with Arm Performix to extract detailed CPU architecture information, cache topology, memory bandwidth, and SIMD capabilities - Config Space (
config_space/) - Defines the optimization search space through a declarative schema, with constraints that respect hardware limitations and model requirements - Bench Runner (
bench_runner/) - Orchestrates model benchmarking by coordinating with llama.cpp for quantized inference and Performix for fine-grained performance collection - Optimizer (
optimizer/) - Implements Bayesian optimization with a Gaussian Process surrogate model to efficiently explore the configuration space - Pareto Front (
pareto/) - Computes and ranks Pareto-optimal configurations to surface the best trade-offs between competing objectives - Exporter (
exporter/) - Converts optimized models to target inference frameworks with deployment-ready artifacts - Report Generator (
report/) - Produces interactive HTML reports with visualizations, metrics, and deployment guidance
The CLI (cli.py) orchestrates the entire pipeline, while the core search loop (optimizer/search_loop.py) manages the iterative optimization process.
Challenges we ran into
Hardware variability complexity - Different Arm cores (Cortex-A53, A72, A76, SVE-enabled variants) have vastly different performance characteristics. Building a configuration space that's meaningful across this diversity while remaining tractable required careful constraint engineering and validation.
Multi-threading instability on ARM platforms - Early optimization runs revealed that llama.cpp's multi-threading support is unstable on some single-socket ARM devices. Configurations with threads > 1 would fail silently, causing entire benchmark runs to be rejected without clear error messages. We addressed this by: (1) adding conservative threading constraints that prevent problematic thread counts on single-socket systems, (2) capturing detailed error information from failed benchmarks to improve observability, and (3) limiting high thread count + high batch size combinations that could exceed memory limits.
Benchmark noise and reproducibility - Extracting reliable performance metrics from real hardware is challenging. System background processes, thermal throttling, and cache contention create noisy observations. We had to carefully tune benchmark isolation and use statistical methods to handle uncertainty in the surrogate model.
Search space scalability - With multiple quantization levels, compiler flags, and inference engines, the configuration space can exceed millions of configurations. Early sampling strategies led to inefficient exploration. We refined the surrogate model tuning and adaptive sampling to improve efficiency.
Framework integration complexity - Integrating with Arm Performix (a proprietary profiling tool) and llama.cpp (a constantly evolving C++ codebase) required careful API adaptation and version management.
Model compatibility constraints - Not all optimization techniques (especially quantization levels) work with all models. Building the configuration space to gracefully handle these constraints without creating infeasible regions was non-trivial.
Accomplishments that we're proud of
✓ End-to-end automation - A complete pipeline that requires only a model, target hardware, and optimization budget. No manual tuning needed.
✓ Production-grade optimization - Configurations are ranked by Pareto optimality and exported to multiple frameworks, ready for real deployment.
✓ Hardware-aware search - The configuration space automatically adapts to detected hardware capabilities, discovering vendor-specific optimization opportunities.
✓ Statistical rigor - The Gaussian Process surrogate model with uncertainty estimates provides principled exploration that balances exploitation and discovery.
✓ Comprehensive reporting - Interactive reports surface not just the best configuration, but the entire Pareto frontier with visualizations, enabling informed decision-making.
✓ Modular, extensible architecture - Each component can be extended independently. New config parameters, hardware profiles, or export formats can be added without touching the core optimizer.
✓ Real-world validation - Successfully optimized actual LLMs on Arm targets, achieving meaningful latency and memory improvements verified through Performix.
What we learned
Hardware expertise is irreplaceable - Understanding the subtle differences in Arm CPU cores, cache hierarchies, and memory systems was critical. Generic optimization is insufficient for hardware-specific gains.
Bayesian optimization is powerful but not magic - The surrogate model is only as good as its observations. Early accurate benchmarks and good prior information matter enormously for search efficiency.
Constraints matter more than the objective - Realistic deployment has hard constraints (memory budget, latency requirements). Pareto optimization that respects these constraints is far more useful than a single "best" configuration.
Reproducibility requires system discipline - Performance measurement on real hardware requires isolation techniques, statistical validation, and careful benchmark design. Otherwise you're optimizing noise.
User experience drives adoption - The friendliest optimization system is worthless if users can't understand or trust its recommendations. Detailed reporting and explainability were as important as the algorithm.
What's next for Arm-Autopilot
Multi-objective refinement - Extend the search to jointly optimize for accuracy, latency, memory, and power consumption, with user-configurable weights for different deployment scenarios.
Distributed optimization - Enable parallel evaluation of configurations across multiple test devices to dramatically speed up the search process.
Incremental learning - Build a knowledge base of optimization patterns from previous runs and projects to seed future searches and reduce required budget.
Advanced quantization strategies - Integrate mixed-precision quantization, per-layer optimization, and activation quantization for more aggressive model compression.
Edge device targeting - Extend hardware profiling to microcontrollers and ultra-low-power Arm cores, supporting truly edge ML scenarios.
Automated model selection - Given a task and deployment constraints, automatically select and optimize the best model architecture, not just parameters.
Community benchmarking - Build a public repository of hardware profiles and optimized model configurations to accelerate the community's adoption of hardware-aware ML.
Log in or sign up for Devpost to join the conversation.