-
Same booklet, only the option order changes. The Yes rate drops from 72% to 5% in Italian, and 21 to 32 of 48 votes flip.
-
Full voting booklet: Apertus says Yes to 71% of proposals in Romansh, against 3 to 7% in German, French and Italian.
-
Asking for a one-sentence reason reduces the order effect but doesn't remove it: Italian still flips on 27 of 48 votes.
What we did
We tested how Apertus v1.5 70B votes on Swiss federal referenda. 48 votes from 2021 to 2026, the official voting booklet in German, French, Italian and Romansh, 5 answers per prompt. We reused the setup of a published Swiss study (Barmettler et al.), so the numbers compare directly with the 66 models they tested.
What we found
- The answer follows the wording. Same booklet, but ask "No or Yes" instead of "Yes or No", and the majority answer flips on 21 to 32 of 48 votes in German, French and Italian. In Italian the yes rate drops from 72% to 5%. Example: the OECD minimum tax (accepted by 78.5% of voters) gets a yes or a no from Apertus depending only on that order.
- Romansh gets a different answer. Same official booklet, same question: with the full booklet and "No or Yes" wording, Apertus says yes in Romansh and no in German, French and Italian on 28 of 44 votes (and 0 the other way round). The gap depends on the setting and is absent in the original study's short "Yes or No" setting. Romansh answers still follow the content, so this is an inconsistency between languages, not noise.
What was hard
Telling the effects apart. At first it looked like Apertus answered differently in every language. Most of that gap in German, French and Italian turned out to be the answer order, not the language. So we ran every prompt in both orders, 5 times each, and only kept differences larger than the run-to-run noise.
How we built it
Python scripts that send the prompts, parse the answers and compute the stats (permutation tests with Benjamini-Hochberg correction). Every number and example in the findings files is generated from the recorded answers.
Next
Run the same tests on the 8B model, and swap the order of the "for" and "against" arguments in the booklet (the "against" chapter is always read last, right before the question).
Log in or sign up for Devpost to join the conversation.