Armillary

Autonomous AI reasoning optimization lab that autotunes and verifies the fastest correct execution path on Arm64.

Comment

Armillary

Inspiration

Optimizing local AI on Arm hardware is still surprisingly manual.

Developers often benchmark one implementation at a time, manually compare results, verify that optimizations did not change the output, and repeat the entire process for every new model or execution strategy.

We wanted to automate this engineering workflow.

Instead of asking "Which implementation should I use?", Armillary asks:

"Can we automatically discover the fastest correct implementation for this hardware?"

What it does

Armillary is an autonomous optimization laboratory for Arm-powered devices.

Given a reasoning workload, it automatically:

  • profiles the hardware
  • benchmarks multiple execution engines
  • verifies exact output parity
  • rejects incorrect optimizations
  • selects the fastest valid implementation
  • generates reproducible benchmark reports
  • creates submission-ready proof artifacts

The project includes an ARC-style reasoning benchmark to provide an objective workload with deterministic correctness.

How we built it

The project is intentionally lightweight and reproducible.

Core components include:

  • Python reference implementation
  • search-space pruning
  • NumPy execution engine
  • native C execution kernel
  • architecture-aware compilation using
clang -O3 -mcpu=native
  • automatic Arm64 hardware detection
  • repeated benchmark measurements
  • median and p95 latency reporting
  • HTML performance dashboard
  • proof artifact generation
  • one-command execution on Apple Silicon

Every optimization must first pass an exact correctness check before it can be selected.

Speed without correctness is rejected.

Measured Results

Measured on an Apple M1 Pro:

Implementation Median latency
Python reference 454.07 ms
Python + Early Pruning 195.07 ms
Native C 3.05 ms
Native C + Early Pruning 1.58 ms

Results:

  • 286.84× throughput improvement
  • 100% exact output parity
  • Arm64 verified

Rather than presenting a single performance number, Armillary measures the contribution of each optimization independently.

Challenges

One of the biggest challenges was making performance claims trustworthy.

Benchmarking is noisy, and faster code is meaningless if it changes the output.

We therefore built the project around three principles:

  1. repeated measurements
  2. exact output parity
  3. reproducible benchmark artifacts

Only candidates that produce identical outputs are considered valid.

What we learned

The biggest lesson was that optimization is not a single trick.

Real performance comes from combining:

  • algorithmic improvements
  • implementation improvements
  • architecture-aware compilation
  • automated benchmarking
  • reproducible validation

Treating optimization as an experiment rather than a benchmark makes the workflow far more reliable.

What's next

The current project demonstrates the optimization workflow using an ARC-style reasoning workload.

Next steps include extending the same optimization framework to real local AI runtimes such as llama.cpp, ONNX Runtime and ExecuTorch while keeping the same reproducible benchmarking and verification pipeline.

We also plan to expand Armillary into a general optimization laboratory for local AI development on Arm platforms.

Built With

Share this project:

Updates