projectNala: LLM Inference on Armv9-A

Inspiration

Modern LLM deployments in the cloud are often bogged down by heavy Python wrappers, bloated container images, and layers of abstraction that obscure the raw capabilities of the underlying hardware. As engineers focused on low‑level systems programming and memory optimization, we wanted to strip away that overhead entirely. The inspiration behind projectNala was to build a bare‑metal, aggressively optimized inference engine that extracts every ounce of compute from Armv9‑A cloud infrastructure without sacrificing developer experience or standard API compatibility.

What it does

projectNala is a high‑performance, ultra‑lightweight LLM inference server designed specifically for Armv9‑A cloud instances. It exposes a fully OpenAI‑compatible REST API (supporting /v1/chat/completions and Server‑Sent Events for streaming) but operates entirely without Python.

By leveraging memory‑mapped (mmap) quantized GGUF models and KleidiAI micro‑kernels, projectNala drastically reduces the memory footprint of running large language models. It directly utilizes Arm's I8MM and DOTPROD instructions to accelerate token generation, making it possible to run massive models on cost‑effective cloud instances with minimal latency.

How we built it

The core backend is engineered entirely in pure C++17.

• Inference Engine – We integrated llama.cpp and configured our CMake build pipeline to ruthlessly enforce the -DGGML_CPU_ARM_ARCH="armv9-a+i8mm+dotprod" flag, ensuring the compiler targeted the Armv9‑A architecture and utilized KleidiAI micro‑kernels.
• API Layer – We used cpp-httplib to construct a lightweight, multi‑threaded HTTP server capable of handling batched prefill and concurrent KV caching.
• Development Workflow – The entire system was developed in a strictly keyboard‑driven Linux environment (Arch Linux, Hyprland, Fish shell). By relying heavily on CMake's EXPORT_COMPILE_COMMANDS, we maintained a perfectly synchronized compile_commands.json for rapid iteration and zero‑latency LSP feedback in Neovim.

Challenges we ran into

• Micro‑kernel Activation – Initially, the build process tried to fall back on standard NEON instructions. Wrestling with the build system to strictly disable x86 code paths and guarantee the KleidiAI micro‑kernels were actually firing required deep dives into compiler flags and architecture constraints.
• Memory‑Safe Asynchronous Streaming – Implementing Server‑Sent Events (SSE) in pure C++ is inherently complex. We had to carefully manage thread states and memory allocation to ensure the server could stream token chunks sequentially back to the client without blocking the main event loop or causing memory leaks.
• Accurate Benchmarking – To prove our optimizations worked, we had to build a custom C++ benchmarking utility (using <chrono>) to measure Time‑To‑First‑Token (TTFT) and resident set size (RSS) in complete isolation from network latency.

Accomplishments that we're proud of

We successfully reduced the operational overhead of serving an 8B parameter model to almost nothing. On an AWS Graviton instance, projectNala achieves a peak memory Resident Set Size (RSS) of just 4.6 GB (down from ~16 GB for FP16), perfectly mirroring the file size of the quantized model. We are exceptionally proud of hitting 23.8 tokens/second during decode and achieving a 13.8 ms Time‑To‑First‑Token on a 512‑token prompt, proving that bare‑metal C++ deployments can vastly outperform standard setups.

What we learned

This project provided an incredible deep dive into hardware‑specific optimization. The core bottleneck in any LLM inference is the General Matrix Multiply (GEMM) operation, formally expressed as:

C=αAB+βC

We learned exactly how Arm's I8MM (Int8 matrix multiplication) instructions optimize the calculation of these quantized weight matrices at the silicon level. Seeing the direct correlation between low‑level math optimizations and high‑level token throughput was a massive validation of systems‑level engineering.

What's next for projectNala

• Continuous Batching – We plan to overhaul the request queue to support continuous batching, which will allow projectNala to handle significantly higher concurrent user loads without degrading the token generation speed of individual requests.
• Dynamic Chat Templates – Currently, the chat template parsing is heavily optimized for Llama 3 / Mistral formats. We plan to write a custom parser that reads and dynamically applies the exact chat template from any GGUF model's internal metadata.
• Arm SME Integration – As newer Arm hardware becomes available in the cloud, we want to expand our build pipeline to support Scalable Matrix Extension (SME) flags for even faster throughput.

projectNala is a testament to the power of going back to the metal—where every cycle counts and every byte matters.

Built With

• armv9
• aws-graviton
• c++
• cmake
• kleidiai
• llama-cpp
• rest-api