ArmForge — On-Device ARM64 LLM Optimization Engine

Open-source Mobile AI engine accelerating on-device LLM inference on ARM laptops using Arm KleidiAI vector kernels & speculative decoding. 100% offline & private with upto +82% throughput boost.

Comment

ArmForge — On-Device ARM64 LLM Optimization Engine

Track 3 — Mobile AI | ARM AI Optimization Challenge 2026

100% Offline · Private · No GPU · No Cloud · Apache 2.0

What ArmForge Does

ArmForge is an open-source optimization engine for running large language models directly on ARM-powered laptops and client devices. It targets the two biggest pain points of local AI: slow time-to-first-token and memory-bottlenecked generation, and it solves both without ever sending a byte of data over the network.

Everything runs on-device: inference is 100% offline once setup is complete, there's no GPU requirement (pure ARM64 CPU inference), no cloud API in the loop, and no telemetry — your prompts and data never leave the machine. The project is released under Apache 2.0.

The Three Optimizations

1. Arm KleidiAI Vector Kernels

llama.cpp ships with generic matrix-multiply code by default, which leaves a lot of performance on the table for quantized models. ArmForge builds against ARM's own hand-tuned dotprod / i8mm kernels instead, routing the model's quantized operations through instructions written specifically for ARM silicon.

cmake -B build -DGGML_CPU_KLEIDIAI=ON -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod

Result: throughput jumps from 20 tok/s to 36.6 tok/s — an 82% improvement, with no change to the model weights themselves.

2. Speculative Decoding

A tokenizer-matched pair of models does the work here: a lightweight 1B "draft" model proposes five candidate tokens ahead of time, and a 3B "verifier" model checks all five in a single parallel forward pass. The output is mathematically identical to running the 3B model alone — only the latency changes.

$$\text{Expected tokens/step} = \sum_{k=1}^{N} k \cdot \alpha^{k-1}(1-\alpha) + N \cdot \alpha^N$$

At an acceptance rate of \( \alpha = 0.75 \) and \( N = 5 \) draft tokens, that works out to roughly 3.2 tokens produced per forward pass.

Result: time-to-first-token drops from 750 ms to 420 ms, a 44% reduction.

3. Hardware Thread-Sweep Auto-Tuning

Rather than assuming a fixed thread count, ArmForge probes the local CPU across \( T \in {1, 2, 3, 4} \) threads using llama-bench and picks whichever setting maximizes memory-channel bandwidth without tipping into thermal throttling or contention.

$$T_{\text{opt}} = \arg\max_{T} \left( \text{tok/s}(T) \right), \quad T \leq T_{\text{physical}}$$

Benchmark Results

All numbers below were measured locally on an ARM64 client device running Llama-3.2-3B-Instruct (Q4_K_M, ~2.0 GB), fully offline and without a GPU.

Configuration Throughput TTFT vs. Baseline Privacy
Baseline (vanilla llama.cpp) 20.08 tok/s 750 ms — 100% offline
+ Arm KleidiAI kernels 36.61 tok/s 620 ms +82% 100% offline
+ KleidiAI + speculative decoding 36.90 tok/s 420 ms +83% 100% offline

Every run is committed as JSON under /results/, so the numbers above are reproducible on any ARM64 laptop — not just ours.

Architecture

ARM64 Client Device (Laptop)
│
├── 1. KleidiAI Vector Kernels (-DGGML_CPU_KLEIDIAI=ON, -b 512)
│    └── dotprod / i8mm instructions → ~2x matrix-multiply speed
│
├── 2. Speculative Decoding Engine
│    ├── Llama-3.2-1B → drafts 5 tokens (fast)
│    └── Llama-3.2-3B → verifies all 5 in 1 pass (accurate)
│
├── 3. Thread-Sweep Auto-Tuner (probes T ∈ {1, 2, 3, 4} cores)
│    └── Selects T_opt → max tok/s without thermal throttling
│
├── 4. Memory Lock (--load-mode mlock, prevents swap-induced jitter)
│
└── 5. Local Dashboard (FastAPI + SSE → localhost:8080)

[ 100% Local · Zero Network · Zero Cloud · Zero GPU ]

Getting Started

git clone https://github.com/Vasanth-repos/Armforge.git
cd Armforge
bash scripts/run_all.sh

run_all.sh handles the whole pipeline end to end:

  1. Installs system dependencies
  2. Builds llama.cpp with KleidiAI enabled
  3. Downloads the Llama-3.2-3B and 1B models
  4. Runs the thread-sweep auto-tuner
  5. Benchmarks baseline vs. optimized configurations
  6. Launches the live dashboard at localhost:8080

Tech Stack

Layer Tool
Inference runtime llama.cpp (KleidiAI build)
ARM acceleration KleidiAI (dotprod / i8mm kernels)
Main model Llama-3.2-3B-Instruct Q4_K_M (~2.0 GB)
Draft model Llama-3.2-1B-Instruct Q4_K_M (~0.7 GB)
Auto-tuner llama-bench thread-sweep
Dashboard FastAPI + SSE (localhost:8080)
Hardware ARM64 laptop (Neoverse / Snapdragon X)
Network None — fully offline

Why This Fits Track 3

ArmForge is built around the constraints that matter most for on-device mobile AI, rather than around raw benchmark numbers alone:

  • Privacy by design — inference never leaves the device, so there's no data-exposure surface to worry about.
  • Genuinely offline — after initial setup, the whole pipeline works with no internet connection.
  • Latency-focused — the 44% TTFT reduction from speculative decoding is aimed squarely at making local models feel responsive, not just fast on paper.
  • ARM-native, not ARM-adjacent — it uses KleidiAI kernels written by ARM engineers for ARM silicon, rather than a generic build that happens to run on ARM.
  • Lightweight footprint — a ~2.0 GB model fits comfortably on most modern ARM laptops.
  • Reproducible — one command rebuilds the entire benchmark suite, with results committed as JSON.

Apache 2.0 · ARM AI Optimization Challenge 2026 · Mobile AI Track 3

Built With

  • arm64
  • bash
  • c++
  • cmake
  • docker
  • docker-compose
  • fastapi
  • gguf
  • huggingface
  • jinja
  • kleidiai
  • llama-cpp
  • mobile-ai
  • numactl
  • offline-ai
  • on-device-ai
  • openblas
  • python
  • pytorch
  • rest-api
  • snapdragon-x
  • speculative-decoding
  • ubuntu
  • uvicorn
Share this project:

Updates

Submission history