NeoServe - Graviton4 LLM Serving Cost/SLO Optimizer

Inspiration & The Problem Everyone Else Skips

Track 2 is flooded with autotuners that maximize single-stream tokens/sec on llama.cpp. But production cloud LLM serving asks a completely different question: at my real traffic concurrency and my latency SLO, which configuration serves tokens cheapest on Arm?

NeoServe answers this exact question by auto-searching vLLM serving configs under real Poisson concurrency, proving each win with Arm Performix PMU hardware top-down profiling, enforcing a quality guard (perplexity budget), and emitting reusable production serving artifacts.

What We Built

• Concurrency-Grid Autotuner: Successive-halving search over memory allocator (mimalloc/tcmalloc), physical thread binding (VLLM_CPU_OMP_THREADS_BIND=phys), KV-cache space, continuous batching tokens, and micro-kernel precision (BF16 → W8A8 → W4A8).
• Quality Guard: Evaluates lm_eval wikitext word perplexity to reject any quantized model that breaches its quality delta budget (≤ 4.0%).
• Arm Performix PMU Hardware Breakdown: Measures hardware top-down instruction retirement (27.5% → 51.8%) and IPC (1.42 → 1.49) proving why the KleidiAI INT4 micro-kernels (kai_matmul_qai8) win.
• Static AArch64 Micro-Kernel ISA Verifier: Disassembles and validates native smmla (Arm i8mm) and bfmmla vector instructions in KleidiAI inner loops.
• Automated Test Suite & Native Arm64 CI: 100% unit test coverage running on native GitHub Arm64 runners with cryptographic SHA-256 ledger integrity.
• Reusable CI GitHub Action (action.yml): Drop-in Action enabling any MLOps team to audit Arm serving SLOs and cost regressions.
• Agentic MCP Interface: Model Context Protocol tool allowing AI assistants (Cursor / Claude) to query recommend_config, get_serving_recipe, and project_cost.
• Pitch-Black Live Dashboard: Production Next.js web application featuring Pareto frontiers, monthly infrastructure cost calculators, and live MCP agent testing.

Real Graviton4 Billboard Evidence (c8g.4xlarge)

• $0.7451 / 1M tokens at p95 SLO (TTFT ≤ 3000ms, TPOT ≤ 200ms) on AWS Graviton4 (c8g.4xlarge Neoverse V2).
• 1.94× Throughput Speedup vs BF16 baseline ($1.4461 / 1M tokens) — 48.5% Cost Reduction ($3,505/mo saved at 5B tokens/mo scale).
• Quality delta: +2.37% PPL (passed ≤ 4.0% budget) — mock: false — 100% SHA-256 Ledger Verified.

Generational Architectural Comparison

• 🏆 AWS Graviton4 (c8g.4xlarge): $0.7451 / 1M tokens (237.5 tok/s, 1,342,077 tok/$) — +58.4% efficiency vs x86
• AWS Graviton3 (c7g.4xlarge): $1.1338 / 1M tokens (142.1 tok/s, 881,989 tok/$)
• AWS x86 Intel Ice Lake (c6i.4xlarge): $1.7904 / 1M tokens (106.4 tok/s, 558,534 tok/$)
• AWS NVIDIA A10G GPU (g5.2xlarge): $1.8276 / 1M tokens (184.2 tok/s, 547,165 tok/$)

1-Command Reproducibility

• Run Full Automated Test Suite: python -m unittest discover -s tests -v
• Verify Cryptographic SHA-256 Ledger: python scripts/verify_ledger.py results/canonical
• Judge Verification CLI: python scripts/demo.py --all
• Verify Micro-Kernel ISA Disassembly: python scripts/verify_kernel_isa.py
• One-Command Production Docker Serve: cd results/canonical/serving_recipe/qwen25-1p5b && docker compose up

Built With

• arm
• aws-graviton4
• docker
• kleidiai
• mcp
• next.js
• onednn
• perplexity
• python
• quantization
• vllm