Lingot

Inspiration

Most projects stop when their model runs on Arm. Lingot starts there.

Optimizing CPU-based LLM inference requires manually testing quantization, runtimes, threads, batching, cost, and quality. We built Lingot to automate that work for Arm cloud servers.

What it does

Lingot takes an open-source model and an optimization target, benchmarks a controlled set of configurations, rejects candidates that violate latency or quality requirements, and selects a Pareto-optimal winner.

It then launches the model through an OpenAI-compatible API and generates a dashboard, reproducible report, and downloadable Arm64 deployment bundle.

How we built it

Lingot uses Python, FastAPI, React, and llama.cpp. We compare four stages on the same Graviton machine:

High precision with the generic runtime
Q4 quantization
Q4 with KleidiAI
Q4 with KleidiAI and Lingot auto-tuning

This separates improvements from quantization, Arm-specific kernels, and Lingot itself.

Challenges we ran into

The hardest challenge was keeping benchmarks fair while measuring latency, throughput, memory, cost, and quality.

We also had to verify that KleidiAI kernels were actually activated, measure complete streamed API requests, and prevent faster configurations with damaged output quality from winning.

Accomplishments that we're proud of

Automated SLO-aware configuration search
Quality-gated Pareto selection
Clear A/B/C/D optimization evidence
Live baseline-versus-optimized token race
OpenAI-compatible serving
Reproducible Arm64 deployment export

What we learned

Cloud inference optimization is multi-dimensional: the fastest configuration is not always the cheapest or accurate enough.

We learned that Arm performance claims are strongest when every optimization layer is measured separately and every result is traceable to raw evidence.