Clusius

Clusius doesn't just suggest migrating to Arm — it migrates, self-optimizes, and proves it live on real hardware: 4x the throughput, 1/4 the cost, zero mocked numbers.

Comment

Clusius — Project Story

Track - Cloud AI

Inspiration

Everyone in cloud infrastructure already "knows" Arm is cheaper, and often faster, than x86 for the right workload. Axion, Graviton, Ampere — the pitch has been repeated for years. And yet almost nobody actually migrates a production AI-inference workload to prove it. Not because the claim is false, but because nobody wants to be the engineer who spent two weeks hand-tuning KleidiAI kernels, quantization levels, and thread counts, only to present a number that turns out to be a cherry-picked slide the moment someone else tries to reproduce it.

That gap — between "Arm is faster" as a marketing line and "Arm is faster, here, on my workload, under my SLA, provably" as an engineering fact — is where Clusius started.

The moment that crystallized it was reading the judging notes for this very challenge. Arm's own judges, in the hackathon webinar, said almost exactly what I'd been circling for weeks: is this a sound engineering implementation, would it still be useful after the hackathon ends, does it actually optimize — make things faster, smaller, smoother — or does it just talk about optimizing. That's not a bar you clear with a benchmark chart in a slide deck. That's a bar you clear by making the tool actually do the migration, actually run the search on real hardware, and actually hand back numbers a stranger can re-derive from a committed JSON file.

So Clusius doesn't recommend a migration. It performs one — end to end, on real Google Axion (C4A) hardware, against a real x86 baseline, with every number traced back to an actual HTTP round trip, never a mock.

What It Does

Clusius is an autonomous agent that takes an x86 AI-inference workload — just a HuggingFace model reference — and runs it through a five-stage pipeline that migrates it to Arm64, searches for the optimal serving configuration on live hardware, and proves the result with a reproducible, schema-validated artifact.

The Five Pipeline Stages

Stage What It Actually Does
1. Analyze Statically scans the workload's build/serving configuration for x86-only assumptions before anything is deployed
2. Migrate Builds two Arm-optimized serving backends on the real target — llama.cpp+KleidiAI and vLLM+oneDNN+Arm Compute Library — from one Dockerfile with a single toggled flag (ENABLE_KLEIDIAI); downloads and quantizes the model to GGUF on the remote target itself
3. Auto-tune Runs a constrained Optuna NSGA-II multi-objective search (maximize throughput, minimize cost) across quantization, threads, batch size, KV-cache precision, and backend — each trial is a real deploy → health-check → benchmark → teardown cycle against the live Arm instance, with the accuracy floor and latency SLA enforced as hard constraints, not suggestions
4. Benchmark Replays the winning configuration's exact traffic profile against the x86 baseline, so the final comparison is never mixed-methodology
5. Report Selects the winning backend with a written justification, renders a human-readable MIGRATION_REPORT.md, and writes a schema-conformant result.json

What a Real Run Looks Like

This is the actual, committed output of a live run against a real C4A + x86 pair — not a projection:

POST /runs  {"model_ref":"Qwen/Qwen2.5-0.5B-Instruct","search_budget_trials":4,...}

  analyze    -> no x86-only blockers found
  migrate    -> built llama.cpp+KleidiAI (arm) + baseline (x86), quantized GGUF on both
  tune       -> trial 0: Q4_K_M, 1 thread   -> 32.9 tok/s, 34188ms p95  [INFEASIBLE: SLA]
                trial 1: Q4_K_M, 2 threads  -> 60.5 tok/s, 15240ms p95  [feasible]
                trial 2: Q4_K_M, 1 thread   -> 32.3 tok/s, 20296ms p95  [feasible]
                trial 3: Q8_0,   2 threads  -> 124.3 tok/s, 6414ms p95  [feasible, WINNER]
  benchmark  -> x86 baseline replay: 30.8 tok/s, 23838ms p95, $0.8734 / 1M tokens
  report     -> MIGRATION_REPORT.md + result.json written

  RESULT: +303.3% throughput · -73.1% p95 latency · -77.0% cost / 1M tokens
  (accuracy: 100.0% on both sides — the constraint held, not just the headline)

Trial 0 being correctly rejected for violating the latency SLA is the detail that matters most: the constraint isn't decorative, it's load-bearing.

Clusius Doesn't Just Recommend — It Builds, Searches, and Proves

Capability What It Does
Dual-backend real Arm build Builds llama.cpp+KleidiAI and vLLM+oneDNN+ACL on the actual target hardware — not cross-compiled, not emulated
Live constrained search Optuna NSGA-II with constraints_func-enforced accuracy floor and latency SLA — an infeasible trial is excluded from the Pareto front, never silently scored as acceptable
Live Pareto-frontier dashboard Every trial streams to the dashboard the instant it completes — winner highlighted, infeasible trials shown in red, bubble size encoding p95 latency
Real, priced cost comparison Cost-per-token uses real Google Cloud on-demand list pricing pulled from the Cloud Billing Catalog API, not a placeholder
Reusable migration templates configs/*.json — swap the model reference and rerun; proven twice already (see below), zero code changes required
Zero-cost replay mode make demo-replay replays committed real evidence through the exact same DB-write and SSE code path a live run uses — full dashboard experience, zero cloud credentials, zero cost
Cost-safe by design SSH target mode needs no cloud credentials at all; opt-in provisioned mode enforces a hard cost ceiling and an always-firing TTL teardown

Proven to Generalize, Not Just to Work Once

The easiest way to fake this kind of project is to run it once, get a good number, and stop. Clusius has been run against two genuinely different models — different architecture families, different size classes — with zero code changes between them, both with real, committed evidence:

Model Architecture Result
Qwen/Qwen2.5-0.5B-Instruct Qwen2 +303.3% throughput, −73.1% p95 latency, −77.0% cost
TinyLlama/TinyLlama-1.1B-Chat-v1.0 Llama +168.6% throughput, −59.6% p95 latency, −65.5% cost

The tuner found a different winning configuration for each — core-pinned with int8 KV cache for TinyLlama, unpinned with fp16 for Qwen — because it's actually searching a live space per model, not replaying a memorized answer.

Architectural Diagram

Clusius System Architecture

How We Built It

Clusius is a uv workspace monorepo with a deliberate separation between the pure engine and everything around it:

  • clusius_core — the pipeline engine itself. Pure Python, zero web-framework dependency, independently testable. Owns the SSH-driven deploy/build/benchmark machinery, the Optuna tuner, and the report generator.
  • clusius_api — FastAPI + SQLAlchemy/Alembic (Postgres) + arq (Redis-backed job queue), streaming stage and trial progress over Server-Sent Events. The pipeline's core run function is deliberately synchronous (Optuna's search loop blocks), so the API bridges it into the async world via asyncio.to_thread and routes the sync callback back to the event loop with asyncio.run_coroutine_threadsafe.
  • clusius_web — a Next.js 15 dashboard that talks only to the API over REST + SSE, never touching the database directly.
  • clusius_agent — a showcase multi-model RAG/MCP agent that serves as a realistic inference workload for Clusius to migrate.

The Arm optimization itself comes down to one Dockerfile and one flag: infra/docker/llamacpp-kleidi.Dockerfile builds llama.cpp from a pinned, verified upstream commit with -DGGML_CPU_KLEIDIAI=${ENABLE_KLEIDIAI}. Flip it ON and you get Arm's KleidiAI CPU matmul kernels; flip it OFF and you get the identical build for the x86 baseline. Same recipe, one flag — so the comparison is never apples-to-oranges.

The search is Optuna's NSGAIISampler in multi-objective mode (maximize throughput, minimize cost), with the accuracy floor and latency SLA wired in through constraints_func — Optuna's own mechanism for excluding infeasible trials from the Pareto-optimal set rather than scoring them as if they passed.

The infrastructure is deliberately cost-conscious: target mode drives an already-running C4A + x86 pair over SSH via Fabric, so Clusius never needs to hold cloud credentials to do the actual migration work. The hosted demo runs on Cloud Run (API, worker, and web, all scale-to-zero or near it), backed by Neon (serverless Postgres) and Upstash (serverless Redis) specifically to keep idle cost near zero, with the real C4A/x86 benchmark pair on Compute Engine — static IPs, tagged firewall rules, and a hard standing rule: the VMs are stopped the instant a test session ends.

dashboard / API call ──▶ FastAPI (Cloud Run) ──▶ arq worker (Cloud Run,
  POST /runs                 persistence + SSE       always-on) ──▶ clusius_core
                              (Postgres via Neon)         pipeline.run_full_pipeline
                                                                │
                     ┌──────────────────────────────────────────┴───────────────────┐
                     ▼                                                              ▼
        SSH target: C4A (Axion / aarch64)                          SSH target: matched x86_64
        • docker build (KleidiAI ON)                               • docker build (KleidiAI OFF)
        • live Optuna trial evaluation                             • baseline benchmark replay
                     └──────────────────────────────┬───────────────────────────────┘
                                                      ▼
                                 MIGRATION_REPORT.md + result.json
                                 (schema-validated, persisted, streamed to the dashboard)

Challenges We Ran Into

1. A real llama.cpp crash that only showed up under live automation

Three real pipeline runs failed in a row with the same symptom: a deployed server that never became healthy within the 180-second timeout. Manually reproducing the exact command by hand on the VM worked every time — which made it worse, not better, to debug. The actual cause only surfaced by SSHing in and running the container without --rm so the crash log wouldn't get garbage-collected: GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) failed. llama-server defaults to 4 parallel slots, and --batch-size 1 — a value our own search space happily offered — is fundamentally incompatible with that default. It only ever crashed when Optuna's sampler happened to land on it, which is exactly why three earlier low-trial-budget runs had gotten lucky and never hit it. Fixed by excluding 1 from the tuner's batch-size search space, with the whole diagnosis left in the codebase's commit history rather than smoothed over.

2. Ephemeral IPs quietly breaking reproducibility

Stopping the VMs between test sessions — the correct cost-safety move — turned out to silently release their IPs. The next run failed with a misleading "Authentication failed," because it was reaching a completely different host that happened to get the old address. Fixed with reserved static IPs, at the cost of a tiny standing fee only while the VMs are stopped (and $0 while they're running).

3. Windows' gcloud + Git Bash path mangling

Any gcloud flag value starting with a single / — like a Secret Manager mount path — gets silently rewritten by Git Bash's MSYS path converter into a Windows path before gcloud ever sees it. No error, just a wrong value discovered only by testing the actual deployed behavior. The blanket fix (MSYS_NO_PATHCONV=1) broke gcloud's own script resolution on this machine; the fix that actually worked was doubling the leading slash, since Linux treats //foo identically to /foo and Git Bash's converter leaves it alone.

4. Missing shared libraries in the Arm Docker image

The first successful docker build produced an image where llama-quantize failed with "error while loading shared libraries" — the Dockerfile had only copied the three named binaries out of build/bin/, not the .so files the server and quantizer dynamically link against at runtime. Fixed by copying the whole directory, matching how upstream does it — something I'd actually read earlier and hadn't fully mirrored the first time.

Accomplishments We're Proud Of

A real, live, end-to-end run — with evidence you can check yourself. Every claim in the README traces to a committed result.json, run-detail.json, and generated MIGRATION_REPORT.md in bench/results/, validated against an open, versioned JSON schema. Not a screenshot. A file you can independently re-run the validator against.

+303.3% throughput, −73.1% p95 latency, −77.0% cost — same model, same quantization, same thread count, same 2-vCPU instance class on both sides, from one build flag Clusius flips automatically and then searches around.

Proven twice, on purpose. A second real run against a completely different model family (TinyLlama, Llama architecture vs. Qwen2) completed cleanly on the first attempt with zero code changes, and the tuner found a genuinely different winning configuration — real proof the search is live, not memorized.

A live Pareto-frontier chart that means something. Winner highlighted, infeasible trials genuinely shown as excluded (not just a green checkmark on everything), streaming trial-by-trial as the search actually runs — because a chart with only ever one color on it isn't proof the constraints work.

Nothing in the chain is mocked, including the boring parts. Building llama.cpp with KleidiAI on real Arm hardware, downloading and quantizing a real model over SSH, running a real load generator against a real server — these involve real infrastructure that breaks in real, unglamorous ways, and the fixes are visible in the repo's own history instead of hidden.

Cost discipline as a first-class feature, not an afterthought. Target mode needs zero cloud credentials to do the actual migration work. Every real test session in this project's history — without exception — ends with the benchmark VMs confirmed stopped.

What We Learned

The bugs that matter most only show up under real load. The batch_size=1 crash was invisible to code review, invisible to unit tests, and invisible to a single successful run — it only surfaced because a fourth automated attempt happened to sample the one bad value, and we chose to reproduce it by hand instead of shrugging it off as flakiness.

A constraint you can't demonstrate being violated isn't proven to work. It would have been easy to only ever show feasible trials on the Pareto chart. Deliberately surfacing the one trial that got rejected — and explaining why — is a stronger proof than a chart where everything is always green.

Reproducibility is a design decision, not a README section. Building a genuine zero-cost, zero-credential path (make demo-replay) to see the exact real-numbers dashboard experience turned out to matter as much as the live pipeline itself — and building it surfaced two more real bugs we wouldn't have caught otherwise.

One good run proves less than two different ones. The Qwen result alone would have been a fine headline number. Running a second, architecturally different model — and having the tuner land on a genuinely different optimum — is what actually proves the pipeline is a general tool and not a script tuned to one lucky model.

What's Next

A head-to-head where vLLM should actually win. Both backends are built and probed on every run, but llama.cpp+KleidiAI has won every trial so far at this small model size and low concurrency. A larger model or a higher-concurrency workload is exactly where vLLM's continuous batching is expected to start winning — that comparison hasn't been run live yet.

Exercising provisioned mode for real. The opt-in path where Clusius creates its own C4A + x86 pair via google-cloud-compute — cost ceiling, TTL auto-teardown, always torn down in a finally block — is implemented and unit-tested but has never driven a real live run. Every real number so far came from an operator-configured target pair.

Bigger search budgets. The committed headline runs used a 4-trial budget to keep real cloud spend small during development — a production migration decision deserves a wider search across the full quantization/threading/batching grid.

Migrating the showcase agent itself. clusius_agent — the multi-model RAG/MCP agent — has been tested standalone but never driven through the full pipeline as the actual migrated workload, which would prove Clusius works on a realistic multi-model serving stack, not just a single model endpoint.

Publishing the optimized artifacts. The quantized GGUF models Clusius produces on the target are currently ephemeral. Publishing them as real, downloadable, reusable artifacts — alongside the migration recipe — turns every future run into a head start for the next team, not just a report.

Built With

Share this project:

Updates

Submission history