Ruhu Operator

Ruhu Operator — Your business operations team, running privately on your laptop.

Comment

Inspiration

A wholesaler in Kano already knows everything needed to answer their hardest question of the week. The sales are in a POS export. The stock counts are in a spreadsheet. The supplier prices are in a WhatsApp thread. What they lack is any way to interrogate it.

Cloud AI serves them badly, for three reasons that have nothing to do with model quality. Connectivity is intermittent and metered. Per-token pricing is impossible to forecast against thin retail margins. And sales, stock, supplier and customer records are commercially sensitive — many owners simply will not send them anywhere.

A model running on the laptop already sitting on the counter removes all three objections at once. That is the whole idea.

What it does

Ruhu Operator reads the records an owner already keeps — POS exports, spreadsheets, invoices, delivery notes, purchase orders, stock reports — and answers operational questions about them:

  • Which SKU is about to stock out, and what to order
  • What this delivery was short by, against the purchase order
  • Whether a supplier is actually cheaper once pack sizes differ
  • What the follow-up actions from Tuesday's meeting were, and who owns them

It runs entirely on an 8 GB laptop with no network dependency, and it never performs a consequential action on its own.

How we built it

The competition scores a GGUF file, not an application. The profiler runs llama.cpp directly against the weights — our LangGraph runtime, typed tools and approval gates never execute during evaluation. Realising this reframed everything: every point comes from the model file alone.

So we built a ruler before making any choice. Twenty corporate_enterprise prompts set in Nigerian SME trade context, scored by 37 deterministic checks — JSON field equality, numeric tolerance, required and forbidden entities, word caps. No LLM judge, so the same weights always produce the same score and two candidates are directly comparable.

python3 eval/run_eval.py --model model/Qwen3-4B-IQ4_XS.gguf --no-think
python3 eval/bench_models.py model/*.gguf --threads 4 --ngl 0

Then we measured on hardware matching the specification — 4 physical cores, 8 GB, Ubuntu 22.04, CPU-only — on both an Intel Xeon and an AMD EPYC, because the rules permit either target CPU.

Three decisions, each reversed by measurement

We nearly shipped a 1.7B. Throughput is capped at 15 tokens per second, so a smaller model looked obviously right: full \(S_{perf}\), better \(S_{eff}\), and surely close enough on accuracy. It was not close. Qwen3-1.7B scored 68.3% against Qwen3-4B's 95.0%, and the failures were substantive. It miscomputes VAT-exclusive amounts. It classifies a payment to a grain supplier as a staff cost. Asked for the most urgent inventory risks, it names a SKU with 75 weeks of cover while missing one at 0.8 weeks.

Since \(S_{acc}\) carries 50% of the score and \(S_{perf}\) only 30%, the arithmetic decides it. Trading down recovers at most 36 points of \(S_{perf}\), worth 10.8 weighted, at a cost of 26.7 accuracy points, worth 13.35 weighted. We kept the 4B and accepted missing the throughput cap — a deliberate trade, not an oversight.

We nearly shipped the wrong quantization. IQ4_XS and Q4_K_M give identical answers on all 20 prompts: the same 35 of 37 checks, prompt for prompt. But peak resident memory differs enormously — 2.32 GB against 4.16 GB — reproduced across four runs on two CPU vendors and two instruction-set builds. The gap far exceeds the 0.22 GB difference in file size. We attribute it to runtime repacking of K-quants for CPU inference, which the IQ path skips.

Writing \(R\) for peak resident memory in gigabytes:

$$S_{eff} = \frac{7 - R}{7} \times 100$$

That is 66.9 versus 40.6 — worth 5.3 weighted points, for identical answers.

We nearly shipped a system prompt that made things worse. Baking a domain instruction set into the GGUF chat template seemed like free differentiation. It cost the 4B 9.2 points. The model began answering plain arithmetic questions in JSON and dropping the working its correct answers depend on. The same prompt helped the weaker 1.7B by 6.7 points. A crutch for a weak model is interference for a strong one.

Challenges we ran into

Our own test prompts were unanswerable. The originals said "using the supplied weekly sales and stock facts" and "from these meeting notes" — context our application injects but the profiler never does. Both were broken by construction, on the most heavily weighted component of the score. Fixing that was the single largest improvement in the project.

Checks that pass wrong answers. One prompt scored as correct because the right SKU name appeared elsewhere in the output. The model had actually flagged an overstocking risk instead of the stockout we asked about. Every check that could pass by coincidence was rewritten to require the underlying reasoning — the computed cover figures, not just the names.

Fine-tuning made it worse. Twice. We built 5,000 training examples with computable ground truth: weeks of cover, invoice totals, variance signs and VAT splits all derive from the same code that emits the document, so there is no label noise and no teacher model. Training loss fell from 1.1847 to 0.1113.

The model scored 80.0% — fifteen points worse than stock. It had learned response-format habits that overrode instruction-following. Every aging example in our data answered in prose, so the tuned model answered "return only JSON" aging questions in prose.

We diagnosed that precisely and rebuilt the generator so each answer renders in a format named in the instruction. Attempt two scored 70.0% — worse still. The fix introduced a new pathology: every JSON example was emitted as an array, teaching the model that JSON output is always a list. On one of our own test prompts, its content is entirely correct but wrapped as [{...}] where a bare object was requested.

Two attempts, both worse, each trading one format pathology for another. We shipped the stock model and wrote down why.

Infrastructure fought back. L4 GPUs were stocked out across every region we tried. Deep-learning images shipped a torchaudio with a mismatched ABI, a jinja2 too old for chat templates, and CUDA present but absent from PATH. One run trained successfully for 90 minutes and lost everything to a missing cmake, because our cleanup logic deleted the VM on failure. We reordered the pipeline so expensive work runs last and failures preserve their artifacts.

Accomplishments that we're proud of

Every claim in our report is backed by a reproducible measurement. The official adtc-profiler on target hardware returned 9.37 tokens per second and 2,396 MB peak resident memory, agreeing with our own harness to within 0.5% — which we did not know would happen until we ran it.

And we rejected our own work three times on evidence. That was harder than building it.

What we learned

Low training loss can measure the wrong thing. A loss of 0.1113 meant our model had learned our generator, not the task. Without a held-out set written by different code, we would have shipped it believing it was better.

Build the instrument before you need the reading. Three confident recommendations were overturned by the harness rather than by reasoning. Without it we would have shipped a 1.7B, with a baked system prompt, and a fine-tune — worse on all three counts.

Knowing when to stop is an engineering decision. Declining a third fine-tune, with the next fix already identifiable, was the right call against a deadline.

What's next

Extending the deterministic tool boundary so that a smaller model becomes viable. If arithmetic, validation and state transitions live entirely in Python and SQLite, the model only has to plan and explain — and a 1.7B that clears the throughput cap becomes reachable without the accuracy cost we measured.

Built with

llama.cpp · GGUF IQ4_XS · Qwen3-4B · Python · LangGraph · SQLite · FastAPI · QLoRA (PEFT / TRL) · Google Cloud Compute · adtc-profiler

Built With

Share this project:

Updates

Submission history