Abstimmungsbüchlein Check

Inspiration

Every Swiss federal vote comes with an official booklet in three languages. Claims about what it says ("the Federal Council recommends…", "the committee argues…") circulate in a different language than the booklet was read in, and nobody checks them against the page. The OST challenge asked for exactly that check: given a claim and the booklet, does the booklet support it, refute it, or not cover it, and where. We wanted a system where the model's answer is always traceable to a page, and where the model is the only thing that judges.

What it does

Two tasks, one container. Task A: booklet PDF + vote name + claim → entailment / neutral / contradiction, with verbatim quotes and 1-based page numbers as evidence. Task B: a given passage + claim → the label. All nine language pairs of de / fr / it work, cross-lingual included. On our held-out validation set: task A macro-F1 0.981 with evidence accuracy 0.988, task B 0.968, at about 1,250 and 820 input tokens per case, with about 20 ms of non-model time per case.

How we built it

Retrieve-then-judge, with the retrieval done by rules instead of similarity. The claim's opening names its source ("Laut dem Bundesrat…", "Le comité estime…"); a small router maps it to one of five parts of the booklet (summary, Federal Council, committee, the text put to the vote, the detailed explanation), a parser finds that part of the right vote from the contents pages and checks every start page against its printed heading, and Apertus v1.5 8B gets that part as numbered paragraphs. It answers with the paragraphs that justify the label and the label, in a JSON schema; the cited paragraphs become the evidence, verbatim, with their page.

Apertus also covers the gaps: it names the part when the rules do not know the opening, reads the contents pages when the parser fails, translates the claim for a BM25 fallback across languages, and takes one focused second look when it answered "neutral" but cited a paragraph (same / different / nothing), which removed most of the "contradiction read as neutral" errors.

Everything local runs on CPU inside the image: pypdf, a 108 MB static multilingual embedding model to rank long parts, BM25. Two worker processes prepare cases while the main process sends requests over one kept connection, so the non-model time the organisers score stays near zero even on a fast server.

We worked in sessions with a fixed method: every change was measured in an interleaved run against the current defaults, adopted only by a rule written down before the run, confirmed on a validation set, and checked on two unseen voting dates. Most of the development was done with Claude Code as a coding assistant; it is not part of the pipeline.

Challenges we ran into

• The full booklet does not work. 37,600 tokens per case and macro-F1 0.858; the routed part scores higher at 1,250 tokens. Finding what to show the model was the whole project.
• Booklet layouts change. The contents page format changed in 2018; some PDFs have unreadable font tables. We tested the parser on every federal booklet from 2016 to 2026 (93 PDFs): 244 of 249 votes pass.
• The endpoint drifts. The public Apertus endpoint changed its answers to identical requests twice in one day and serves two backends. We stopped comparing runs across time and only compare arms inside one interleaved run.
• An 8B model's neutral bias. When the booklet gives a different number than the claim, the model prefers "not stated" over "contradiction". Five prompt rewrites did not fix it; a narrow second call did.
• Task B's other-ballot errors. Unrelated passages about a similar subject get called contradictions. Six attempts (ballot-naming fields, label renaming, title-prefixed retrieval, a relevance second call, a cross-lingual tournament, BM25 selection) either traded errors for other errors or broke the answer format. We report them as negative results.
• Scoring rules that count our own time. Non-model wall time and tokens are scored relative to the best team. We replaced the ONNX embedding model with a static one (2.3 GB → 200 MB, 0.13 s → 0.016 s per case) and made the container answer every case under 512 MB.

Accomplishments that we're proud of

• Macro-F1 0.981 / 0.968 on validation and 0.990 / 0.977 on two unseen voting dates, with the 8B only, zero-shot.
• Every label comes from Apertus. Python wrote 1 of 1,948 labels in the final runs (a neutral fallback for one unreadable answer).
• Evidence that matches the gold passage 98.8 % of the time, always with its real page.
• Stability: four full runs of the 600 dev cases gave the same 600 labels.
• A technical report with an ablation from "no booklet" (0.435) to the final system, an error analysis that separates routing from reading errors, and a table of about twenty ideas that did not pass their rule, with numbers.

What we learned

• Against a model's systematic bias, repeating the question does not help; changing the question or the input does. Two calls to the same 8B agree with each other 97 % of the time, including when both are wrong.
• Exact words beat similarity for keeping unrelated text out: a BM25 fallback with the claim translated by Apertus answered "neutral" correctly where embedding retrieval found something that merely sounded related.
• Pre-registered adoption rules are what kept us honest at 0.97: with 300-case dev sets, ±0.02 is noise, and most "improvements" vanish on validation.
• [If 15B produced a 70B row: "The 70B scores x.xxx on dev with the same pipeline, +0.0xx over the 8B; the gap is smaller than the gain from choosing what the model reads."]

What's next for Booklet-Grounded NLI

• Fix the remaining task B error class with a relevance check that does not confuse "different ballot" with "opposite claim"; our attempts failed on the 8B.
• A browser extension or chat front end that takes a claim from a news article and returns the booklet page.
• Cantonal voting booklets, which follow different layouts.
• [After session 16: the sealed test result, run once after the freeze: task A x.xxx, task B x.xxx.]

Built With

• apertus
• bm25
• natural-language-inference
• onnxruntime
• openai-api
• pypdf
• pytest
• python
• retrieval-augmented-generation
• sentence-transformers
• swiss-ai