Consylr

Inspiration

Reconciliation is the daily reality of enterprise finance teams across Africa: settlement fees deducted at source, withholding applied before remittance, batched settlements against itemised invoices, and references mangled in transit between mobile-money and bank rails. The work is repetitive, the records are confidential, and the costliest failure mode of any assistant is a confidently invented explanation. We wanted a model where an honest “I can’t tell from this evidence” is a first-class, measured behaviour—and where confidential ledgers never leave the machine.

What it does

Consylr reads a bounded packet of financial records, policy lines, and documents, then returns one structured JSON answer: which records correspond, which differences are exceptions and why, what remains unmatched, and what a human should do next.

Every identifier it cites must resolve to an item in the supplied packet. When a difference is real but the packet does not explain it, Consylr reports cause_not_established and requests the missing evidence instead of inventing a cause.

That behaviour is measured, not asserted. Across 75 packet pairs that differ by one policy line, the shipped model flips in the correct abstention direction 77.3% of the time and records zero false refusals on the settled halves. It runs fully offline on ordinary CPU hardware and peaked at 3.4 GB in the official audit environment.

How we built it

A procedural data factory generates reconciliation cases whose answers are correct by construction: every claimed cause is provably present or absent, and every case is validated, budgeted, and de-duplicated by a writer that refuses invalid output rather than merely warning.

We fine-tuned Qwen3-4B-Instruct with LoRA on approximately 12,000 synthetic cases, evaluated it on a sealed 204-packet exam plus a 75-pair twin probe, and quantised it to GGUF Q6_K using an importance matrix calibrated on the training distribution. Every reported score sits beside a null control, and the shipped artefact’s provenance is traceable by digest from the public download back to the training corpus.

Parsing and arithmetic remain deterministic. The model handles correspondence and evidence judgement inside a bounded packet. A human authorises every consequential action.

Challenges we ran into

Quantisation was capability-affecting rather than uniformly lossy. Our first 4-bit build retained matching accuracy—F1 0.884—while silently destroying the refusal behaviour: the probe flip rate fell to 1.3% from 84% unquantised. The standard exam did not expose that loss; only the purpose-built minimal-pair probe caught it. A training-distribution importance matrix recovered the behaviour at Q6.

Accomplishments that we're proud of

The tuned model reaches 45.6% exact match on the held-out sealed set, compared with 0.0% for its base model, and every identifier it cites resolves to the packet. Its refusal behaviour survives quantisation at a 77.3% abstention-direction flip rate, with zero false refusals on settled probe halves.

The audit-mode rerun reconciles cleanly: every measured field is within 0.9%, and memory is within 0.1%. Four exam versions and two artefact candidates were rejected by our own gates before anything shipped.

What we learned

Aggregate metrics flatter. A model can score well on abstention while never reading the evidence, and a quantisation can preserve correspondence F1 while deleting the behaviour a product is built around. The instruments that caught both—a positional null control beside every reported F1 and minimal-pair probes—cost less than the mistakes they prevented.

What's next

We plan pilot deployments against real reconciliation workloads while preserving the product boundary established here: deterministic parsing and verification, model-led interpretation inside a bounded packet, and humans authorising every consequential action.

Built With

• gguf
• imatrix-quantisation
• llama.cpp
• lora
• python
• qwen3