Apertus on Swiss political questions

Inspiration

Apertus is built to be Switzerland's trustworthy, multilingual and sovereign LLM. Its own Apertus Charter asks it to stay neutral on contested political questions (Art. 3.7, 7.4), to reason consistently across interactions (2.7), to keep the same substance whatever the tone (3.4), to support plain language (3.6) and to respect linguistic diversity (3.5).

In a direct democracy, that promise matters. People increasingly ask chatbots about ballot questions. A Romansh speaker, someone who needs plain language, or a user writing in Swiss German should not get a different political answer than a German speaker writing formally. We did not ask which side Apertus takes. We asked whether its answers stay the same when only the form of the question changes.

What it does

We put Apertus through the 75 policy questions of the Swiss voting advice application smartvote (2023 questionnaire, fixed answer options). We vary only things that should not matter:

6 languages: German, Swiss German, French, Italian, Romansh, English
3 tones: original wording (benchmark), plain language (CEFR B1) and colloquial
4 query modes: one question per context; all 75 questions in one conversation; all 75 questions in one prompt; and a Charter round, with the full Apertus Charter loaded as system prompt
Order: original order, seeded permutations, and a swap test that exchanges only the first questions
Answer options: forced choice, and with smartvote's «no answer» option
Models and reasoning: Apertus v1.5 8B and 70B, thinking on and off

Because greedy decoding on a shared endpoint is not bit-reproducible, every condition is repeated up to 33 times. Every effect is measured against this noise. In total: about 400 runs and 399,000 answers.

Our core stability metric counts side changes: questions whose mean score lies on opposite sides of 50 in condition $ c $ and in the benchmark $ b $, counted only on questions that are stable (at least 90 % identical answers) in both:

$$ \text{SideChanges}(c) = \sum_{q \in Q_{\text{stable}}} \mathbb{1}\left[ (\bar s_{q,c} - 50)(\bar s_{q,b} - 50) < 0 \right] $$

where $ \bar s_{q,c} $ is the mean smartvote-coded score (0 = No, 100 = Yes) over the repetitions.

How we built it

Data: the official smartvote 2023 questionnaire in DE, FR, IT, RM and EN, used with the explicit permission of smartvote.ch. Thanks to the smartvote team!
Translations and rewrites: Claude translated the questions into Swiss German, and the team validated every question. Apertus 70B itself generated the plain-language and colloquial variants, following one shared, language-independent rule set. Two independent open-weight reviewer models (Gemma 4 31B, GLM 5.3) checked meaning, tone and answer options, followed by a repair and a candidate round. Humans validated every case where the reviewers disagreed. DE, CH, EN and FR are validated; IT and RM follow for our semester thesis.
Pipeline: a model-agnostic Python package for any OpenAI-compatible endpoint, configured via LLM_NAME, LLM_BASE_URL and LLM_API_KEY. It renders prompts with the pinned Apertus chat template and stores the full Apertus-format conversation, raw output and reasoning of every call; parsing is a separate step. Checkpoints and resume, pinned model revisions, fixed seeds, automatic quality checks, and lossless compression to keep the data under 100 MB.
Infrastructure: a local CPU VM serving Apertus v1.5 8B for development, and the CSCS Inference API (vLLM) for all reported runs. Everything ships in Docker: make run reproduces all five issues on a clean checkout in about 20 minutes and reports PASS or NOT REPRODUCED per issue.
Analysis: documented Jupyter notebooks with a noise baseline from identical repetitions, excess over the noise, side changes on stable questions, Wilcoxon and exact McNemar tests over questions with Holm correction, and bootstrap confidence intervals. The thresholds of make run are calibrated by simulating it 2,000 times from the stored runs.

Challenges we ran into

Noise: greedy decoding over a shared API is not deterministic. A single run cannot separate a real effect from chance, so we repeated every condition and tested against the measured noise.
Compute: the 8B model runs at about 0.5 tokens per second on our CPU VM. Moving to the CSCS Inference API made the runs about 200 times faster.
Six languages, one meaning: keeping rewrites faithful, especially in Swiss German and Romansh, required strict rules (same content, same answer options, no added judgement), reviewer models and human validation. Prompt instructions stay in English, so only the question and its options change language.
What counts as an answer: separating a deliberate no stance from a refusal and from an invalid output took parsing rules for each language.
Licensing: we cleared the reuse of the questionnaire with smartvote before building on it.

Accomplishments that we're proud of

An end-to-end red-teaming harness that anyone can point at the next Apertus release (or any other model) and rerun with one command.
A validated, multilingual political test set in the national languages plus Swiss German, with plain-language and colloquial variants and a documented data specification.
The Charter round: to our knowledge the first test of whether Apertus behaves more neutrally when its own Charter sits in its context window.

What we found

Five issues, all reproduced automatically by make run:

1. The first answers set the position of the whole conversation. A different question order changes 14–65 of 75 answers beyond the noise (8B). Swapping only the first three questions drops extreme answers on the other 69 identical questions from 77 % to 16 % (German). 70B does not adopt the style, but its positions still move with the order.
2. Same question, other national language, opposite position. On 37 of 75 stable questions, 8B agrees in one national language and rejects in another (70B: 28). Terminating the bilateral agreements with the EU is supported only in Swiss German, in both models.
3. Plain and colloquial wording change the answer. Validated rewrites with the same meaning change 14–29 of 75 answers (8B, German). Adding only «Findest du,» («Do you think,») turns «spend much more» on social welfare into «spend much less». 70B shows up to 17 side changes in Swiss German.
4. Charter neutrality fails in Romansh. With the Charter as system prompt and a «no answer» option, 8B abstains on 46–75 of 75 questions in five languages, but on none in Romansh, and whether it abstains at all is decided by its first answer.
5. Reasoning loops. With thinking on, 8B reasons until the token limit on 17–31 of 75 questions and gives no answer, mostly on budget questions.

What we learned

Measuring political bias in an LLM depends heavily on the instrument, and a single run says little. With repetitions and an explicit noise baseline, effects that looked like chance became clear and reproducible. The more interesting question was stability, not position: Apertus answers the same question differently depending on who asks, in which language, in which words and in which order.

What's next

Validate and run the remaining languages (Italian, Romansh) for our semester thesis.
Extend the 8B / 70B comparison and add reliability measures across all conditions.
Share the results with smartvote and the Swiss AI Initiative.

Built with

Python · Docker · vLLM (CSCS Inference API) · Hugging Face Transformers (Swiss AI fork) · FastAPI · Proxmox · Jupyter · pandas · NumPy · SciPy · Matplotlib · uv

Team Härter Chern: Ivo Kohlert, Leon Wink, Reto Steinmann (ZHAW)