ParetoPilot

Compare Arm64 inference configurations with reproducible benchmark reports.

Comment

Inspiration and project overview

I built ParetoPilot as a solo Track 2: Cloud AI project because one speedup number is not enough to choose an inference configuration. A useful decision also needs latency, memory, quality, reliability, and enough evidence for someone else to reproduce the result.

ParetoPilot is an Arm64 inference benchmarking and decision tool. It can recommend an optimized configuration, but it can also keep the reference configuration when an improvement is not convincing. It separates model choice from server sizing, records why each candidate passed or failed, and uses the same decision process in local reports and CI.

What ParetoPilot does

ParetoPilot checks where each measurement came from, removes configurations that miss the chosen quality or resource limits, and finds the best tradeoffs (the Pareto frontier). It then selects a configuration for the chosen goal. A 1% cutoff defined before the run means a tiny numerical lead does not automatically justify more complexity.

Each run can produce recommendation.json, machine-readable decision details in decision-passport.json, a Markdown decision summary in optimization-receipt.md, report.html, and a CI result. These show the selection, rejected candidates, failed constraints, source hashes, and alternative priorities. ParetoPilot refuses malformed inputs, missing measurements, and synthetic data presented as measured evidence. It also does not invent a deployment command when the study does not provide one.

What I optimized on Arm

I tested Qwen2.5 1.5B Instruct on a GitHub-hosted Ubuntu 24.04 Arm64 runner with four Arm Neoverse-N2 vCPUs. GitHub Actions built a pinned llama.cpp revision in generic and KleidiAI-enabled variants.

I tested four isolated stages:

  1. Q8 generic reference
  2. Q4 quantization
  3. Q4 with the KleidiAI-enabled build
  4. Q4 with KleidiAI and a 512-token micro-batch

The published v1.1 model and latency study used the mirrored order A-B-C-D-D-C-B-A, giving every candidate an early and late pass on the same runner. I measured prompt and generation throughput with llama-bench, behavior and streamed latency with llama-server, and peak memory with GNU time -v. This made it possible to distinguish quantization, the Arm-specific build change, and runtime tuning instead of presenting them as one opaque before-and-after result.

Measured results

The v1.1 latency study is GitHub Actions run 30055662526, preserved in release v1.1.0. For the single-client p95 end-to-end latency goal defined before the run, ParetoPilot kept the Q8 generic reference at 2231.933 ms. The 1% cutoff was 2254.2522 ms, and no Q4 candidate entered that shortlist. The tuned Q4 configuration used substantially less memory, but it was 3.40% slower on that goal, so Q8 remained the latency choice.

I then ran a separate v1.4 capacity study, run 30144901854, to answer a different question: how many server slots and simultaneous clients should each candidate use? I tested two candidates across server parallelism 1, 2, and 4 and client concurrency 1, 2, and 4 in mirrored forward and reverse passes. All 288 measured requests completed with zero recorded failures.

Candidate at its selected point Generation p95 TTFT p95 end-to-end Peak RSS Quality
Q8 generic at P4/C4 84.94 tok/s 756.6 ms 3056.8 ms 3448.1 MiB 21/24
Q4 + KleidiAI + 512 micro-batch at P4/C4 90.67 tok/s 639.9 ms 2852.6 ms 2031.3 MiB 20/24

At P4/C4 for each candidate, tuned Q4 measured 6.74% higher generation throughput, 15.42% lower TTFT, 6.68% lower end-to-end latency, and 41.09% lower peak RSS. P4/C4 means four server slots and four simultaneous clients. Its quality score was 95.2% of the Q8 score. This makes it a useful serving alternative, while Q8 remains the published latency result.

How I built it

I wrote the core as a dependency-free Python 3.12 package. It validates benchmark sets, constraints, llama-bench output, server evaluations, load results, resource measurements, and multi-candidate manifests. I versioned the published archives and locked them with SHA-256 checksums.

The CLI includes one command that verifies both published studies:

paretopilot verify-published --output-dir output/published-proof

It downloads the two pinned ParetoPilot release archives, verifies their exact sizes and hashes, and safely extracts them. It then reproduces all nine published v1.1 comparisons and both capacity artifacts and writes published-proof.md and published-proof.json. It works on Linux, macOS, or Windows and does not rerun inference or claim current hardware performance.

I also packaged the decision process as a GitHub Action so another project can fail CI when measured evidence is missing, a constraint is violated, or the selected configuration changes unexpectedly.

How to run and validate it

Clone the Apache-2.0-licensed repository and install it with Python 3.12 or newer:

git clone https://github.com/agrovr/ParetoPilot
cd ParetoPilot
python -m pip install -e .
paretopilot verify-published --output-dir output/published-proof

For a fresh native Arm64 measurement, fork the repository into a public GitHub repository with Actions enabled. In the Actions tab, select Native Arm64 candidate study, choose Run workflow, use the default main branch, and keep 10 repetitions. The workflow runs on ubuntu-24.04-arm, builds the pinned generic and KleidiAI-enabled variants, executes the complete measurement sequence, and uploads the resulting artifacts. The equivalent GitHub CLI command is:

gh workflow run candidate-study-arm64.yml --ref main -f repetitions=10

Challenge, learning, and impact

One exploratory run completed its measurements but was rejected because its server logs did not expose the exact KleidiAI marker I had declared as required evidence. I increased log verbosity and made that setting part of the verified command. That taught me that enabling an optimization is different from proving that it ran.

I also learned that optimization sometimes means keeping the baseline. ParetoPilot’s published latency result is a no-change decision, while the separate capacity study shows where tuned Q4 is useful. ML and platform engineers can use the same approach to compare configurations without hiding quality loss, memory cost, failed constraints, or an inconclusive result.

Limits and next steps

These measurements cover one model family and workload on one Arm64 runner. The capacity matrix is a bounded fixed-concurrency study, not an open-loop production benchmark, cost study, energy study, or MLPerf result. Each cell has a small sample, so I report observed results rather than statistical significance. The KleidiAI log marker records what llama.cpp reported; it is not kernel-level acceleration proof.

Next, I would repeat the study on more Arm64 machine types, models, and workloads, then add longer open-loop tests while keeping each result versioned and replayable. Arm Performix could add optional profiling context, but it would remain separate from the benchmark evidence.

Primary links

Built With

Share this project:

Updates

Submission history