MustafaHaAp

Inspiration

Apertus is meant to stay neutral on contested questions and to treat Switzerland's languages fairly. Most red-teaming checks a model with one question at a time, usually in English. Real use looks different: conversations run over many turns, apps and agents feed earlier messages back in, and people say all sorts of things about what the model told them before. We wanted to see whether Apertus keeps its neutrality under those everyday conditions, and whether it behaves the same in German, French, Italian and Arabic as it does in English.

What it does

We tested all four Apertus v1.5 models (8B and 70B, each with and without thinking) on contested history and politics and on ambiguous questions where the fair answer is "we can't tell". Every question has a clear neutral answer, so we can score the replies automatically and compare them with controls.

The short version: the models are neutral when asked directly, but that neutrality is easy to lose. Small, ordinary-looking changes to the conversation shift them to one side of a contested question in most cases, across all four models and all five languages. When asked to justify themselves, they often back it up with confident historical "facts" that turn out to be wrong when checked against sources. Similar effects show up on stereotype questions, more strongly in some languages than in others, and the smaller model is noticeably less neutral outside English even without any manipulation.

We submitted four findings with full details to the organisers.

How we built it

We wrote a small Python toolkit with no outside dependencies. It talks to the hosted Apertus endpoint, parses answers in five languages, and runs the statistics. We ran a multilingual baseline of about 2,000 generations, then built a probe that compares several conversation setups against controls. Free-text answers were labelled blind against a written rubric, and quotes were checked automatically against the original replies. A single make run rebuilds every number from the stored evidence inside Docker.

Challenges we ran into

Our first idea didn't work: persuasion-style prompts made no measurable difference, so we changed direction. The open-model judges we planned to use weren't available to us, so we had to rethink judging. One model variant puts its reasoning inside the answer text, which our parser misread at first; we caught it and re-scored. Working in five languages also means translations need checking by native speakers, which is still on our list.

Accomplishments that we're proud of

Every model and every language we tested is affected, and the effect holds up against controls over more than a thousand calls. We fixed our statistics before the main run, kept the null results, and every number in the report can be rebuilt with one command.

What we learned

Neutrality that only holds for a clean, single question is fragile. Bigger models and "thinking" variants weren't automatically more robust. Language matters: the same model can behave quite differently depending on the language you ask in. And it pays to decide how you'll score things before looking at the data.

What's next for MustafaHaAp

More questions and repeated runs, native-speaker review of the translations, and human judges for the free-text answers. We'd also like to share concrete suggestions with the Apertus team on making neutrality hold up in longer, multilingual conversations.

Built With

• gemma
• local-llm
• python
• unittest