TinyLLM-ARM-Pro: LLM Inference for the Other 6 Billion
What Inspired This
I'm a second-year ECE student from Tamil Nadu, India. I don't have a GPU. I don't have a cloud budget. I have a MacBook Air with an Apple M4 chip — and a question that wouldn't leave me alone:
Why does every AI tool assume you have a $10,000 GPU cluster?
94% of the world's developers work on ARM devices — phones, cheap laptops, Raspberry Pis. The entire inference ecosystem is built for NVIDIA hardware. I wanted to prove that was unnecessary. Not by claiming it. By building it.
What I Built
TinyLLM-ARM-Pro is a five-layer ARM inference optimization pipeline built entirely from source on Apple Silicon — no downloaded binaries, no pre-packaged models, no GPU.
Layer 1 — Quantization Pipeline
I compiled llama.cpp v9660 natively for ARM64:
cmake -B build -DGGML_METAL=ON -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
Then converted TinyLlama 1.1B and Phi-2 2.7B through our own pipeline:
Key result: Q4_K_M achieves 6.6× inference speedup over FP32 at 68% less RAM, validated on WikiText-2 (PPL 8.73 ± 0.05).
Layer 2 — Hand-Written NEON SIMD Kernels
I wrote three progressive versions of ARM matrix multiply in C using NEON intrinsics:
// One NEON FMA instruction — 4 float32 multiply-accumulates at once
sum_vec = vmlaq_f32(sum_vec, a_val, b_vec);
The v2 kernel uses 4 independent accumulator registers to break the FMA dependency chain:
$$\text{Speedup} = \frac{t_{\text{naive}}}{t_{\text{NEON v2}}} = 12.52\times$$
All results verified correct against the naive scalar baseline.
Layer 3 — ARMv8.6-A I8MM (SMMLA) Kernels
I implemented the vmmlaq_s32 instruction — the same instruction
llama.cpp uses internally for quantized inference:
// 32 INT8 multiply-accumulates in ONE instruction
acc = vmmlaq_s32(acc, a_vec, b_vec);
With pre-packed weight layout (matching GGUF's storage format) and 4-tile blocking, the tiled kernel achieves 12.48× speedup over naive INT8 scalar code.
Layer 4 — Native Inference
Using Flash Attention, Metal GPU full offload, and quantized KV cache:
$$\text{Prompt processing} = 1329 \text{ tokens/sec on Apple M4}$$
Layer 5 — Adaptive Inference Planner
A decision engine that detects hardware, recommends the optimal quantization config, explains every tradeoff, and validates its own recommendation live:
python pipeline/run_all.py --auto --model tinyllama
# → Detects Apple M4, 17.2GB RAM
# → Recommends Q4_K_M (fastest within 40% RAM safety margin)
# → Runs live inference → Speed error: 3.5% ✅
What I Learned
The most important lesson had nothing to do with ARM.
Early in the project, I measured a 73.9% quality improvement over a community reference model. It looked like a huge win. Instead of publishing it, I ran WikiText-2 — the academic standard.
The real result: our Q4_K_M (PPL 8.73) is statistically equivalent to the reference (PPL 8.74). The 73.9% was measurement noise from a flawed pseudo-perplexity method.
I corrected it. Publicly. In the report, in the README, in every file it appeared in.
That moment taught me more than any benchmark: honesty is an engineering discipline, not just a value.
I also learned:
- NEON FMA dependency chains are the primary bottleneck at small sizes
- Metal GPU warmup costs must be excluded from steady-state measurements
- Subtracting two noisy timing measurements doubles the noise (almost destroyed my stress test methodology)
- llama.cpp's GGUF pre-packs weights for I8MM layout — I had to rediscover this the hard way when my first I8MM kernel was slower than naive code
How I Built It
Week 1: Set up environment, established FP32 baseline (19.0 tok/s), built quantization pipeline from source.
Week 2: Wrote NEON FP32 kernels (v1, v2). Wrote I8MM SMMLA kernels (packed, tiled). Ran academic WikiText-2 validation. Caught and corrected the 73.9% false finding.
Week 3: Fixed all JSON output plumbing. Built the Adaptive Planner.
Wired run_all.py --auto end-to-end. Added Phi-2 2.7B as second model
to prove model-agnostic pipeline.
Week 4: Cross-device validation on GitHub Actions ARM64 cloud runner (Cobalt 100 processor). Stress testing. Research report. This submission.
Hardware: Apple MacBook Air (M4, 17.2GB unified memory). No GPU. No cloud. No team.
Challenges
The I8MM kernel. My first implementation was slower than naive code
at large sizes. Diagnosis: the vmmlaq_s32 instruction requires a
specific interleaved memory layout that I was constructing at runtime
(memcpy inside the hot loop). The fix — pre-packing the weight matrix
once at initialization — is exactly what llama.cpp does in production.
I learned this by reading the source code, not from documentation.
Methodology bugs. The stress test's first version showed decode
speed increasing 3× as prompt length grew. That's physically impossible.
Root cause: llama.cpp's KV cache was silently reusing the first call's
computed prefix for the second call, making the subtraction meaningless.
Fixed with explicit model.reset() between calibration calls.
The 73.9% correction. Admitting a headline finding was noise and correcting it publicly, mid-project, was uncomfortable. I did it anyway. It made the submission stronger.
Cross-Device Validation
Benchmarks run on two independent ARM64 machines:
| Device | NEON v2 Speedup | I8MM Tiled Speedup |
|---|---|---|
| Apple M4 (local) | 12.52× | 12.48× |
| GitHub Cobalt 100 (cloud) | 12.52× | 10.14× |
All correctness checks passed on both devices. ✅
The Mission
"LLM inference for the other 6 billion. No GPU. No cloud. Just ARM."
This project was built in Tamil Nadu on a laptop. Every result is reproducible from the open-source code. Every claim is backed by a real benchmark. Every mistake was caught and corrected publicly.
That's what democratized AI engineering actually looks like.
Built a complete quantization pipeline from source on ARM64 — not downloaded pre-built models, but compiled llama.cpp natively and quantized every model ourselves. Our Q4_K_M is statistically equivalent to community references (PPL 8.73 vs 8.74) — proved by WikiText-2 validation, not assumed.
Wrote and verified three generations of hand-written ARM NEON SIMD kernels in C — naive → v1 (4×) → v2 blocked (12.52×) — every version numerically verified correct against the scalar baseline.
Implemented the ARMv8.6-A SMMLA instruction (vmmlaq_s32) — the same instruction llama.cpp uses internally — from scratch, diagnosed why the first version was slower than naive code, fixed it with pre-packed weight layout, then pushed further with 4-tile blocking to 12.48×.
Caught and publicly corrected our own false headline finding (73.9% quality improvement → actually statistical equivalence). This took courage but made the project credible.
Validated on two independent ARM64 machines: Apple M4 and GitHub Actions Cobalt 100 cloud runner — NEON v2 achieves 12.52× on both.
Built a working Adaptive Inference Planner that detects hardware, recommends a quantization config, and validates its own prediction live — 3.5% speed prediction error on TinyLlama, 11.3% on Phi-2.
Proved the pipeline is model-agnostic: Q4_K_M is optimal on both LlamaForCausalLM (TinyLlama) and PhiForCausalLM (Phi-2) — same pipeline, different architectures, same finding.
What's next for TinyLLM-ARM-Pro markdown
Raspberry Pi 4 validation — real measured numbers on a $35 constrained device (1GB RAM) to fully prove the "any ARM device" claim with empirical data, not architectural reasoning.
Third model family — add Qwen2.5-0.5B (a Qwen architecture) to the cross-model leaderboard. Three independent architectures all showing Q4_K_M optimal would be a publishable finding.
AWS Graviton3 benchmarks — server-class ARM validation using the free tier Oracle Cloud ARM instance to show the pipeline works across the full ARM spectrum: edge (M4) → cloud (Graviton).
OpenAI-compatible API layer — wrap the Adaptive Planner in a FastAPI server with a
/v1/chat/completionsendpoint so any developer can drop TinyLLM-ARM-Pro in as a local, offline, zero-cost replacement for cloud LLM APIs.One-line installer —
pip install tinyllm-arm-prothat auto-detects the device, downloads the right model, and runs the planner. Zero configuration for the developer.CO₂ measurement — quantify the energy saved per inference request vs cloud GPU. "X grams of CO₂ saved per 1000 requests" is a concrete impact metric that tells the mission story in numbers, not just words.
The long-term goal: become the reference implementation that any developer with an ARM device uses to run LLMs locally — the way llama.cpp is the reference for C++ inference, TinyLLM-ARM-Pro becomes the reference for ARM-first, privacy-first, budget-first deployment.
Log in or sign up for Devpost to join the conversation.