Cascade4Arm - CloudAI Track

Inspiration

Every team deploying a small LLM today ends up choosing between two bad trade-offs: self-host it yourself on whatever hardware you have (usually a stock x86_64 box, full precision, zero optimization), or call a hosted API and hand over latency, cost, and control to someone else's GPU. Neither path takes advantage of what a model this small actually needs to run well. We wanted to know: if you optimize specifically for Arm — Graviton4, Neoverse-V2, KleidiAI kernels — and add a router smart enough to not waste a big model on an easy prompt, how much of that tax can you actually claw back? The answer surprised us enough that we built a live head-to-head to prove it instead of just quoting a number.

What it does

Cascade4Arm is a two-model routing system — Qwen2.5-0.5B and Qwen2.5-1.5B — served on AWS Graviton4 with llama.cpp built specifically for Neoverse-V2, using Arm's KleidiAI kernels (SVE2, i8mm, bf16) and q4_K_M quantized weights. A lightweight router classifies each incoming prompt: factual and everyday queries go straight to the 0.5B model for sub-50ms time-to-first-token, while reasoning, code, and analysis prompts automatically escalate to the 1.5B model with speculative decoding enabled, so quality never gets traded away for speed.

The app has two modes:

Compare — send one prompt to three stacks simultaneously (our cascade, a self-hosted x86_64 baseline running Qwen2.5-3B full precision, and a hosted API baseline running Gemma 4 26B with reasoning) and watch them race live, with full metrics (TTFT, tok/s, cost, route taken) reported per run.
Benchmark — a fixed 50-prompt suite run against both AWS boxes at concurrency 1, 2, 4, 8, and 16, reporting throughput, p50/p95 latency, and error rate at every step, all wall-clock measured on the actual instances.

How we built it

Model layer: Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct, quantized to q4_K_M.
Inference engine: llama.cpp compiled for aarch64, using Arm KleidiAI kernels to hit SVE2 / i8mm / bf16 instruction paths on Graviton4's Neoverse-V2 cores.
Routing: a classifier in front of the two models that scores prompt difficulty and picks a route (0.5B-direct vs 1.5B + speculative decoding), so the small model handles the bulk of everyday traffic and the larger model is only paid for when it's actually needed.
Baselines for comparison: a matched self-hosted baseline (same vCPU/RAM class, same model family, Qwen2.5-3B, full bf16 precision, Hugging Face + PyTorch on stock x86_64) for a controlled apples-to-apples comparison, plus a hosted API baseline (Gemma 4 26B-A4B via OpenRouter/Google AI Studio) as a real-world reference point.
Benchmark harness: custom load generator sweeping concurrency 1→16 over a fixed 50-prompt suite against both AWS boxes, recording per-request TTFT, tokens/sec, total latency, and errors, exportable as CSV.
Both EC2 instances are public, anonymous, and rate-limited server-side to keep the demo safe to expose.

Challenges we ran into

Getting a fair, controlled comparison — matching vCPU count and RAM between the self-hosted baseline and the Graviton4 cascade so the ratio reflects the software/hardware optimization, not just "one box has more resources."
The hosted API's free tier is rate-limited per minute, which meant a full 50-prompt benchmark sweep against it would spend most of its time absorbing 429s rather than measuring anything real — so the Benchmark tab intentionally only compares the two boxes we control, and the hosted API stays as a single-prompt reference point in the Compare tab instead.
Tuning the router's classification so escalation to the 1.5B model (and speculative decoding) only kicks in when it meaningfully helps output quality, without either over-escalating (losing the speed advantage) or under-escalating (losing quality on hard prompts).
Getting KleidiAI's SVE2/i8mm/bf16 kernel paths actually engaged (rather than falling back to generic aarch64 codepaths) required care in how llama.cpp was built and which quantization format was used.

Accomplishments that we're proud of

20.6x faster than a matched, same-hardware-class self-hosted baseline.
4x faster end-to-end, 150x faster time-to-first-token (15ms vs 2.24s) against a real hosted API.
At saturation (concurrency 16), p95 latency of 4.35s vs 38.6s on the baseline, with zero errors on either stack across the full benchmark sweep.
Output cost of $1.31 per million tokens — no GPU, no over-provisioned box, just Arm CPU cores used well.
A router that provably sends the right prompts to the right model — the per-prompt breakdown shows 12–20x speedups on factual/everyday prompts and 5–6x on reasoning/code/analysis prompts that correctly escalate.

What we learned

Most of the "optimization" available in small-model serving isn't in the model at all — it's in not calling the big model when you don't have to, and in making sure the hardware you're already paying for is actually being used (correct kernels, correct quantization, correct instruction paths) instead of running generic, unoptimized code on it. Arm's Neoverse-V2 cores plus KleidiAI kernels closed most of the gap people assume only a GPU can close, for models at this size.

What's next for Cascade4Arm

Extend the cascade to three tiers and benchmark against more hosted APIs under fair rate limits.
Add automatic router calibration from the benchmark suite instead of hand-tuned thresholds.
Package the whole stack as a one-command deploy for any Graviton instance, so other developers can reproduce these numbers on their own AWS account in minutes.