Inspiration
Swiss citizens vote on federal proposals several times a year, and each household receives an official booklet with the arguments of both sides. More and more people ask a chatbot instead. We wanted to know: does Apertus, built as a trustworthy model for Switzerland, give voters both sides or does it simply confirm the opinion they bring?
What it does
ADuo red-teams Apertus v1.5 (8B and 70B) on 20 Swiss federal votes. It measures whether answers cover the official pro and contra arguments equally, whether a stated yes or no opinion steers the answer (sycophancy), how complete and correct the answers are, and which way the model leans when forced to answer "Ja" or "Nein". Every judgement uses an external reference – the official voting booklet and the Swissvotes dataset, never our own opinion.
How we built it
A seeded sample of 20 votes (2016–2024), 189 key arguments extracted from the booklets and reviewed by both of us, and mirrored prompts generated in code: 2,120 Apertus responses. Our main metric is a sycophancy score:
$$S_{syc} = B_{pro} - B_{contra}, \quad B = \mathrm{Cov}_{pro} - \mathrm{Cov}_{contra}$$
The analysis plan was pre-registered with Git tags, and every deviation is logged. A tested Python package reproduces all numbers from the stored responses with one make run in Docker. We used Claude as a coding and writing assistant; all coding decisions were ours.
Challenges we ran into
None of the five LLM judges we tested reached our pre-set agreement threshold with human coding (κ ≥ 0.70), so we hand-coded 120 answers ourselves, blind to model and condition – under a hard deadline.
Accomplishments that we're proud of
A fully pre-registered, openly documented study; a measurement free of our own political judgement; and five findings that anyone can reproduce and check against verbatim Apertus answers.
What we learned
A good external reference is everything when measuring balance. Pre-registration keeps you honest when things go wrong. And automatic evaluation is the bottleneck: an LLM judge should be validated like a second coder, not trusted as the instrument.
What's next for ADuo
More answers per condition, the milder framings, the same votes in different national languages, and a systematic check of stated results against Swissvotes.
Log in or sign up for Devpost to join the conversation.