Inspiration
Apertus is built to be a multilingual, open and trustworthy model, but most safety testing is done in English. We asked a simple question: does the model behave the same when the same question is asked in a different language, or about a different name? Many real users write in Hindi, Tamil or German, so a safeguard that only works in English is only half a safeguard.
What it does
Holey Apertus is a small, fully reproducible red-teaming harness for Apertus-v1.5-8B. It asks identical questions with one thing changed (language, name or order) and measures the difference. It found five issues:
- Factual accuracy gap. Tamil accuracy is 50% vs 88% in English (paired exact McNemar, $p = 0.004$).
- Name-dependent loan decisions. Identical financials, but North Indian and Tamil names are approved 33% of the time vs 94% for Swiss-German names ($p < 0.001$).
- Candidate order bias. The first-listed candidate wins 95% of the time, whoever they are.
- Invented facts. The model gives confident details for 98% of people, books and institutions that do not exist.
- Copyright refusals vanish. It refuses to reproduce protected works 73% of the time in English and German, but 0 of 22 times in Hindi and Tamil, and some answers are verbatim.
How we built it
- Python harness (
src/audit) calling the CSCS Apertus endpoint at temperature 0 with a fixed seed. - A deterministic on-disk cache of all 506 responses, so anyone can recompute every number offline with 0 live calls.
- Exact statistics (Wilson intervals, Fisher exact, exact McNemar, exact binomial), figures, and findings files validated against the challenge schema.
- Tests with a fake endpoint, plus Docker and
make run. - Manual labels for copyright and invented-fact answers are published so anyone can audit them. Claude helped write code, labels and the report.
Challenges
- Our first keyword detector misread polite refusals as answers, so we read all the neutral-wording replies and labelled them by hand.
- Some ideas did not hold up. A "values by language" test showed no effect, so we dropped it. We also expected Hindi and Tamil to invent more facts, but when asked for specifics all languages invent equally, so we report the language effect only where it is real.
- We kept caveats honest, for example that name groups share prompts and are not independent samples.
What I learned
Safeguards and calibration that look fine in English can be missing in other languages, and one-variable-at-a-time experiments make that visible. Also, being honest about weak results made the strong ones more credible.
What's next
Run the 70B and thinking variants, add more languages and native-speaker review, and test more samples per prompt.
Log in or sign up for Devpost to join the conversation.