Inspiration
Knowledge work in African enterprises runs on documents — HR policies, service contracts, procedures — and the tools that could answer questions about them assume a cloud, a subscription and a permanent connection. All three are assumptions, and in much of the continent all three fail.
The user we built for is an office worker with the laptop they already have: 8 GB of RAM, integrated graphics, intermittent connectivity. They do not need a general assistant. They need a correct answer to "what is the reimbursement deadline", with the paragraph it came from, without sending an internal contract to a third party.
One constraint shaped everything: an assistant that is confidently wrong about a policy is worse than no assistant, because a wrong figure is acted upon.
What it does
LeanLM answers questions about your own documents, entirely offline, on commodity hardware. Ask it something and it runs a fixed ten-phase pipeline: assess the machine's current resources, compute a context budget from them, select the passages worth their tokens, assemble a grounded prompt, generate, then check the answer against the passages that were actually sent.
Three behaviours matter more than speed:
Every sentence is traceable. Answers cite labelled excerpts, and the
citation is verified against the real label set — a fabricated [S7] is caught.
It refuses. When no passage clears the relevance floor, retrieval returns nothing and the system says the documents do not allow a conclusion. On our evaluation set this is correct 100% of the time.
Offline is enforced, not promised. While a request runs, non-loopback sockets and DNS resolution raise an error and the attempt is recorded. There is no telemetry endpoint to disable, because there is none.
Measured on a 4 vCPU / 8 GB Ubuntu instance with Qwen3.5-2B (Q4_0, 1.16 GB, Apache-2.0), against our own ground-truth set of 12 probes:
| answer accuracy | 90% (10 answerable probes) |
| refusal accuracy | 100% (2 probes the corpus cannot answer) |
| source accuracy | 100% — cited the document that states the fact |
| hallucination rate | 0% |
| generation throughput | 13.25 tok/s, cv 0.03 |
| model footprint | 1.16 GB of a 7 GB budget |
How we built it
Eight capabilities, each owning consecutive stages of one pipeline, each in its own package with the same seven modules — contracts, models, policies, services, telemetry, benchmarks, README. A tool walks capability → package → contract → tests → benchmark → documentation and fails the build on a gap, so the architecture cannot quietly rot into a diagram.
Some decisions worth naming:
BM25 rather than embeddings. An embedding model is a second model resident in RAM. On an 8 GB laptop already holding a quantized LLM, a 400 MB encoder competes with the thing the user came for. Retrieval runs in under 4 ms with no model loaded.
A figure makes a passage unique. "Reimbursed within 30 days" and "within 45 days" are 96% lexically identical and mean different things. Deduplication compares numeric signatures first, so it can never delete the correct answer and keep a plausible wrong one.
Context capped at 4096 tokens regardless of what the model advertises. The KV cache scales with context, not with weights; a large advertised window is a model-card feature and a memory bill.
Zero mandatory dependencies. Every third-party library is optional with a standard-library fallback, because a machine with no working package index is the machine we are building for.
Challenges we ran into
The hard part was not building it. It was discovering that four consecutive measurements described our tooling rather than our system, and each one produced a plausible number.
A simulated backend scored 100%. We had a stand-in for development. Left unchecked it would have shipped throughput figures no model ever produced. It is now labelled on every artifact and refused by the packaging gate.
A scratchpad scored 83%. The model reasons before answering; with a small token budget it produced twelve unfinished drafts. Our scorer searched for the right figure, found it inside the draft, and marked eight probes correct. A truncated generation can now never be scored correct, in either direction.
Our own cleaner mangled the answers. The code that stripped an echoed prompt searched the whole output for any prompt line. A reasoning model quotes its own instructions while working, so the search hit mid-thought and removed the first half of the response — including, in several cases, the answer itself.
The same model scored 8% and 92% on the same machine, in the same hour. The
only difference was the backend. Recent llama-cli builds are a chat
application: they apply the chat template, which puts a hybrid reasoning model
into thinking mode, and wrap the output in a banner and an echo of the prompt.
Through llama-server the same model answered cleanly. Throughput told the same
story from the other side: cv 0.01 against cv 0.66.
What broke the pattern each time was printing the evidence next to the score — the answers, the phase, the configuration. The score alone was never enough to notice.
Accomplishments that we're proud of
The refusal works. Asked for Tesla's share price against a corpus of HR policies, the same model without our layer answered:
240 EUR
An invented price, grounding 0.0, four times out of four. LeanLM declines. That corpus fits entirely inside the context window, so this is not a retrieval trick — the model simply invents when nothing stops it.
The comparison is honest about where we lose. Our benchmark reports the rows where the naive path is faster, because a comparison that can only flatter us is not a measurement.
476 tests, covering the foundations, the contracts, each capability, the runtime, the full pipeline and the delivery gate. The ones we value most assert honesty: that an invented answer is caught, that a fabricated citation is detected, that an out-of-corpus question is refused, and that simulated results cannot be submitted.
What we learned
A measurement without its conditions is not a measurement. We lost most of a day to a profile that had been edited but not selected — 8% where the intended configuration gave 92%, with nothing on screen to tell the two runs apart. Every result now prints its profile, backend and model on the first line.
Rankings do not always transfer between machines. We told ourselves that
relative comparisons survive a change of hardware. They do when the scoring
function is monotone. min(TPS/15, 1) is flat above the reference: on a slow
laptop the smaller model wins on throughput, and where both saturate the term
cancels and accuracy decides. The ranking does not shift — it inverts.
Cross-platform is not a nice-to-have for something whose premise is the laptop you already have. Three defects were invisible on Linux and only appeared on a Windows machine.
What's next for LeanLM
A larger evaluation set: twelve probes is small, and one threshold in the off-topic filter was tuned while looking at them. That is overfitting risk, plainly stated.
Grounding by n-gram overlap catches invented figures and fabricated citations. It does not catch a fluent answer that recombines real passages into a false claim. That is the next honest problem.
And the French corpus, which the same pipeline answers without a single configuration change, deserves a real multilingual evaluation rather than a demonstration.
Log in or sign up for Devpost to join the conversation.