wennab

Project Overview

Challenge track: Mobile AI. wennab targets on-device inference on Arm client hardware — laptops, phones, tablets, boards — where quantisation choices decide whether a model runs at all. Every measurement in this submission was taken on an Apple M1 (Arm64), 8 GB, CPU only.

Created during the challenge period. The repository, all four tools, the 25 tests, the registry and the case study were written in August 2026, inside the submission window. The measurements they report were taken on the same machine over the preceding days.

wennab is a set of four command-line instruments for anyone quantising or recalibrating a model to run on an Arm laptop, phone or single-board machine. It does not quantise anything and does not run a model — you keep llama-quantize and lm-evaluation-harness. It exists for the part in between, where optimisation results quietly turn into fiction.

An optimisation report almost always reads: "our build scores 0.68 against the baseline's 0.67 at the same size." Three things are usually wrong with it, and none is visible in the number.

The two files differ by more than the thing under test. I rebuilt a 2 B model with my own importance matrix and it came out 23 MB heavier than the file I meant to beat. It would have been easy to credit my calibration for the accuracy difference. A tensor-by-tensor check showed the extra weight came from somewhere else entirely — the reference's publisher had used a different type map, dropping attn_qkv to IQ4_XS and lifting ssm_out, ssm_alpha, ssm_beta, on every layer except those congruent to 3 modulo 4. Two causes, one number, and no measurement taken afterwards could have separated them.

The calibration corpus contains the exam. A matrix calibrated on a text protects the weights that text activates. Put your evaluation prompts in the corpus and the rebuild improves its own marks without improving the model. Writing the corpus yourself makes this more likely, not less: templates and test prompts come from the same hand, the same register, the same example cities. Mine collided on eight consecutive words. I only knew because I checked.

Two totals are not a comparison. The standard error of a 0.67 score over 200 questions is 3.3 points. A four-question gap is inside the noise, and reporting it as an improvement is the most common way a null result gets published as a win.

Functionality & Output

wennab twin — is this even a valid pair? Reads the tensor type map off two GGUF files and says whether they differ by anything other than their values.

$ wennab twin reference.gguf candidate.gguf
4 type group(s) differ — these files are NOT a valid pair

  tensor                   candidate → reference  layers          bytes
  attn_qkv.weight               Q5_K → IQ4_XS    18 of them    -35,389,440
  ssm_alpha.weight            IQ4_XS → Q8_0      18 of them       +313,344
  ssm_beta.weight             IQ4_XS → Q8_0      18 of them       +313,344
  ssm_out.weight              IQ4_XS → Q5_K      18 of them    +11,796,480

  difference +22,966,272 B

Then replay the reference's map, so the two builds differ by one thing only:

$ wennab twin reference.gguf --emit types.txt
$ llama-quantize --imatrix yours.imatrix --tensor-type-file types.txt \
      source-BF16.gguf candidate.gguf IQ4_XS

$ wennab twin reference.gguf candidate.gguf
identical type maps (320 tensors)
  difference +0 B

That second run is what makes everything downstream mean something.

wennab corpus — a calibration corpus you can reproduce and read. Generates a corpus for a register from hand-written templates under a fixed seed. No real documents, so nothing unpublishable; no scraped legal text, so no licensing fog. It prints 4-gram diversity rather than assuming it, because that is the ceiling on what the method can do: hand-written text is a fixed quantity, so lengthening the corpus only repeats it, and a matrix estimated on repetition over-weights whatever repeats. Measured on mine — 330 kB → 0.387, 240 kB → 0.439, 180 kB → 0.495 — so I shipped the shortest of the three.

wennab guard — refuse a corpus that contains its own exam. Exit code 1 on a shared 8-gram, so it drops into a pipeline. Eight consecutive words do not coincide between independent texts even in the same register — and the longest shared run is printed either way, because a clean pass at n=8 does not tell you whether you cleared it by a mile or by one word.

wennab paired — compare question by question. Reads the samples_*.jsonl that lm_eval --log_samples already writes, pairs by document id, runs an exact McNemar test.

$ wennab paired runs/reference runs/candidate
200 questions, paired by document id

  reference   134/200  (0.670)
  candidate   136/200  (0.680)

  agree       196/200  (98%)
  reference only right : 1
  candidate only right : 3

  p = 0.625

  Consistent with chance. 4 question(s) separate these
  models — report that number, not the difference of the totals.

Paired by document id rather than by position: two runs can order their documents differently, and pairing by position then compares unrelated questions while looking perfectly healthy.

The case study: it told me my own work was worthless

These tools were not written to prove a point. They were written because I was recalibrating a 2 B model's importance matrix for francophone enterprise documents on an 8 GB M1, and could not tell whether it had worked.

With twin I built a control matching the reference to 32 bytes — the length of a filename in the metadata — with an identical peak RSS across three runs. With guard I found and removed a real eight-word collision between my corpus and my own test set. Then paired gave the answer:

196 of 200 answers identical. p = 0.625.

At equal size, speed and memory, recalibrating the importance matrix did not change what the model decides. I shipped the original weights and published the null result. Full protocol, all four builds and the reasoning behind each control are in case-study/, with per-question outcomes in case-study/results/ so the table can be recomputed rather than believed.

An honest null result took a day to establish and would have taken ten minutes to fake. That gap is the whole reason this repository exists.

Setup Instructions for Arm

Everything above was built and measured on an Apple M1, 8 GB, CPU only (-ngl 0) — an Arm machine under exactly the constraint these tools are for.

pip install git+https://github.com/benewende-dev/wennab
wennab --help

# or from a clone, no install:
git clone https://github.com/benewende-dev/wennab && cd wennab
python -m wennab.cli twin reference.gguf candidate.gguf
python -m pytest tests/          # 25 tests, well under a second```

Python 3.11+. `gguf` is the only dependency and only `twin` needs it — the other three read plain
text and JSON, deliberately, so they run on anything. Nothing here compiles and nothing here calls
a GPU, so it is portable to any Arm64 Linux as-is. `twin`, `guard` and `paired` each run in well
under a second and hold nothing large in memory, so they belong on the device rather than on a
workstation you have to keep in sync with it.

Three practical notes from that machine, none of which I expected:

- **Compute the matrix from BF16, but know the price.** Requantising an already-quantised file
  stacks two roundings and measures that instead of your change. But 3.8 GB of BF16 does not fit in
  8 GB of RAM alongside anything else: `llama-imatrix` ran at **12 % CPU** — waiting on the SSD, not
  computing — 12.6 s per chunk, 18 minutes for 88 chunks. On this class of machine the bottleneck is
  paging, not precision. Starting from Q8_0 (2 GB, fits) is the first thing to try.
- **Measure with nothing else running.** Closing a browser moved swap from 9.3 GB to 992 MB on the
  same machine. Any throughput number taken without doing that measures your desktop.
- **Three runs, take the median.** I recorded 34.3 t/s and 31.2 t/s on the *same file* two weeks
  apart on a fanless laptop. Compare builds measured the same day, alternating, or you are measuring
  thermal drift.

## What I learned

That the hardest part of an optimisation is not making the change — it is proving the change is what
you measured. Three separate traps almost got me on a single 2 B model: a type map I did not choose,
an exam sitting inside my own calibration corpus, and a four-question gap I nearly reported as a
gain. Each one would have produced a plausible, publishable, wrong number.

I also learned that an 8 GB Arm laptop is an honest teacher. When `llama-imatrix` sits at 12 % CPU
because it is waiting on the SSD, you find out very quickly which of your assumptions came from
reading and which came from measuring.

## What's next for wennab

- **More registers.** `enterprise-fr.toml` is one worked example. The registry format is plain TOML,
  so medical, legal or agricultural registers are a file, not a code change.
- **A tighter loop on device.** `twin`, `guard` and `paired` already run in under a second on the M1;
  the goal is a single `wennab check` that takes two GGUFs and two eval runs and answers the only
  question that matters — *is this difference real?*
- **The case study, repeated where the lever should bite.** My null result is on IQ4_XS, where four
  bits per weight leave room to be wrong in. At Q2_K or IQ1_S there is far less precision to
  allocate, so allocating it well should matter more. That is the experiment I want to run next —
  and this time the instruments exist before the experiment does.

Le champ About the project est complet. Reste sur la même page, plus bas :

Built with — tape-les un par un (Entrée après chaque) :
python
llama.cpp
gguf
arm64
apple-silicon
quantization
lm-evaluation-harness
pytest

« Try it out » links :
https://github.com/benewende-dev/wennab

Built With

• apple-silicon
• arm64
• gguf
• llama.cpp
• lm-evaluation-harness
• pytest
• python
• quantization