-
-
The 13th AHV pension passed with 58.25% yes. Apertus 1.5 70B says "rejected" in all five national languages.
-
Initiatives the government lost: right 17%. Referendums it lost: right 67%. The model defaults to "rejected".
-
Share of answers saying "accepted": 55% in German, 33% Italian, 5% French, 2% English. True share: 46%.
-
On votes the government lost: 95.5% right when given the numbers, 30.3% when asked to recall the outcome.
-
Report figure: the language effect holds across every run, with and without thinking.
Inspiration
Apertus is Switzerland's open model, and the federal vote is Switzerland's defining civic act. In a quick check, Apertus 1.5 70B said the 13th AHV pension (accepted 3 March 2024 with 58.25% yes) was rejected, in every national language. We wanted to know if that was a fluke.
What it does
Scrutineer asks Apertus 1.5 about every Swiss federal vote since 1990: 334 decided votes plus 4 that had not been held yet, 6,716 questions in German, French, Italian, English and Romansh. Eight question families probe each vote: outcome, yes share, the government's recommendation, cantonal majority, a reasoning test that gives the numbers but hides the title, the constitution, a not-yet-held vote, and a one-sentence summary.
Findings:
- Our pre-registered claim, that Apertus repeats the government's line on votes the government lost, failed: the gap is 8.8 points, under our 10-point falsifier.
- The real pattern is a default answer of "rejected" (73% of answers against a true rate of 54%). On initiatives the government lost it is right 17% of the time; on referendums the government lost it is right 67%, more often than when the government won.
- The default depends on the language: about 95% "rejected" in French and English, close to the truth in German. The four languages agree on only 135 of 309 votes. Reversing the answer options does not change French or English.
- Reasoning works, recall fails: 95.5% with the numbers given, 30.3% from memory, on votes the government lost.
- Serving matters: the same official 70B weights on two providers differ on 7.7% of answers at temperature 0. Public AI's 8B endpoint answers word for word like the 70B, so we report no 8B result.
How we built it
Gold answers come from Swissvotes (Universität Bern), pinned by sha256. Titles are the Federal Chancellery's official wording from its linked-data service, in all five languages, and nothing was translated by a model. A deterministic five-language parser (150 test fixtures) grades most answers, and a local open-weights judge (Qwen2.5-7B on llama.cpp, never Apertus) grades the rest without seeing which model answered. The claim was written to a ledger before the first request. make run rebuilds the dataset, replays all grading and verifies every number in Docker in under a minute, with no API key.
Challenges we ran into
Public AI's 70B endpoint returned HTTP 429 for hours, so we moved the run to the official 70B weights on Featherless through the Hugging Face router and kept the first 287 answers to measure how much two providers disagree. Comparing outputs showed that the "8B" endpoint answers like the 70B. The human check exposed a judge bug ("rejected by a majority" read as a contradiction), so we fixed the prompt and re-graded everything, reporting both versions. A runner bug ignored our pre-registered sampling order for the thinking run; we fixed it and documented it rather than hiding it.
Accomplishments that we're proud of
A benchmark that reports a failed claim honestly and finds something more useful instead. Every number in the report, README, site and data card is generated from one results file and checked by hash. Every deviation from the plan is logged (26 decisions).
What we learned
Check that an endpoint serves the model its name claims before comparing model sizes. Report the cross-provider flip rate next to any single-run accuracy. A human check only measures something if the human labels before seeing any model's labels.
What's next for Scrutineer
Replay the official 8B weights locally to settle the 8B question, add short everyday names for votes alongside the official titles, get native-speaker review of the templates (especially Romansh), and share the finding with the Apertus team.
Log in or sign up for Devpost to join the conversation.