Apertus-Recht

At a rally her words are punishable; to her sister they are not. With no law or the statute pasted in, Apertus 70B convicts both in German. 1,980 twin cases, DE/FR/IT, verdicts computed from the law.

Comment

Inspiration

A woman denies Tamils equal worth as human beings at an authorised rally before about 300 people. Under Art. 261bis of the Swiss Criminal Code (discrimination and incitement to hatred), that is punishable. She writes the same words in a message only to her sister: not punishable, because the statement is not public (BGE 130 IV 111). Without the law in the prompt, Apertus 70B answers "Ja, strafbar" to both.

Anyone can ask Apertus whether their own words are punishable, in German, French and Italian, with or without the law retrieved (Apertus v1.5 8B: 698,641 downloads worldwide in the 30 days before 9 October 2026). So we asked which law Apertus applies, not how often it is right.

What it does

Apertus-Recht is 1,980 scenarios in German, French and Italian, built as minimal twins: pairs that differ in exactly one legal fact (a family dinner instead of a public meeting, a canton instead of a religion, 2019 instead of 2021). Code computes every verdict from the statute on Fedlex (pinned by sha256) and the leading cases BGE 130 IV 111 and 131 IV 23. The verdicts match the outcome of all 57 in-scope Swiss court rulings (52 distinct cases), both acquittals included; a rule that always convicts would match 55. Where the law's verdict flips between twins and Apertus's does not, the pair names the element it misapplies.

To our knowledge, no dataset before this one tests which version of a real statute a model applies, element by element, in the three official languages.

Per language: 660 scenarios in 220 twin groups (430 punishable, 215 not, 15 contested: settings the law leaves open), each with an acceptance criterion, a chosen and a rejected answer, a settled or contested label, its legal source and its derivation.

What Apertus v1.5 70B does (exact forced choice from log-probabilities, German, Ja/Nein):

  • No law in the prompt: it acquits 0 of 215 lawful scenarios. Asked the reverse ("would a court acquit?"), still 0: not a yes-bias.
  • The rules in plain sentences: it acquits 107 of 215. It applies publicity and the exclusions the prompt names, but in German and Italian still convicts when the target is a canton, a disability or a political opinion, and convicts acts from before 1995.
  • One sentence on the 2020 extension backfires: it convicts 22 of 125 punishable sexual-orientation cases, against 296 of 305 for race, ethnicity and religion. It does not learn the date; it takes sexual orientation off the list.
  • The statute pasted verbatim, as a retrieval pipeline would: back to 0 (French 12, Italian 43). The statute's text does the damage, not its length: before the same rules, unrelated legal text of equal length leaves 36 of 75 acquitted, the statute 11 (McNemar p = 6·10⁻⁸).
  • The rules make it convict lawful criticism of a religion: of 20 lawful criticisms of a religion (a separate dignity family), it convicts 6 without the law and all 20 with the rules.
  • Retellings in the first person, natural paraphrases and the 8B show the same failure; a second statute (Art. 179ter) fails differently.

How we built it

  • statute-twins, a Python package (Apache-2.0, 128 tests, CI): a statute is five YAML files; a 100-line engine computes the verdicts; the generator refuses any twin that changes more than one fact. Art. 179ter was added the same way.
  • Apertus 70B through Public AI: exact forced choice from log-probabilities through the endpoint labelled apertus-v1.5-8b (docs/builds.md gives the evidence that it serves the 70B), free text through apertus-v1.5-70b. The 8B as the public GGUF on CPU in GitHub Actions.
  • Labels checked five ways: blind annotation of 960 items by Claude agents (all 939 settled items agree with the code), Qwen as a model of another family (910 of 939; each of its 29 misses convicts where the law acquits, the errors Apertus makes), agent adjudication (0 of 94 sampled items wrong), mutation tests, and 57 Swiss court rulings.
  • Contamination: 0 of 35,241 core-item 13-grams (scan of 9 October) occur in the 1,069,365 decisions of voilaj/swiss-caselaw, an Apertus v1.5 pre-training source; the positive control finds 846 of 1,525 13-grams of the statute.
  • Free text: two Claude judges (κ 0.993) and Qwen-SEA-LION (κ 0.92) judge every German answer against its acceptance criterion, and a verdict parser agrees on 698 of 700 answers with a clear verdict; French and Italian are judged the same way; the seeded 20% sample is released for audit.
  • Pre-registered and frozen by hash. The locked held-out bar (a 15-point gain in every framing, per rung) is met trivially at R0 by every model and, at R3, only by the 8B in Italian; the 70B reaches 14.8 (German and Italian). Post hoc, at R3 the law read off the twins predicts 0.94–0.98 of the 70B's held-out verdicts. Everything beyond the locked test is labelled exploratory, and 27 dated amendments record every later change.

Challenges we ran into

  • Exact verdicts for a real statute: no retroactive punishment, synonyms on the closed list, settings that are genuinely open. Contested cases get their own label instead of a forced verdict.
  • Separating the law from the format: reversed questions, unrelated legal text of equal length, ablations of the pasted law, and a mirror answer that flips only the verdict word.
  • Privacy: the fact summaries we wrote for the dignity court check are pseudonymised data on criminal proceedings; we release their codings and hashes, not the summaries.

Accomplishments that we're proud of

  • To our knowledge, a new kind of test: one-fact twins with placebos (changes that must not flip the verdict), real amendment dates (1995, 2020) and verdicts computed from the pinned law, in German, French and Italian. Compared with LEXam, LawShift, TALD, OpenExempt and Multilingual HateCheck in docs/prior-art.md.
  • make run (Docker, CPU, no key, a few minutes) rebuilds every scenario and label, re-scores every stored response and recomputes 105 claims behind the numbers in the README and report; CI runs it on every push.
  • Every Art. 261bis scenario outside the 162-item exploration battery carries a note fitted to the case: on lawful ones, that "not punishable under Art. 261bis" does not mean "allowed" and where to turn; on punishable ones, where to report.
  • Open by default: dataset CDLA-Permissive-2.0 with per-item licences and provenance, code Apache-2.0, legal texts public domain. The scenarios are synthetic; no data was collected from people. Consent template with an AI-use clause; PII scan 0 hits.

What we learned

  • Without the law, Apertus ranks punishable cases above lawful ones (AUC 0.72–0.77; 0.5 is chance), but sets its threshold so far toward "punishable" that almost no lawful case falls below it.
  • Plain rules beat the pasted statute.
  • In German, even when the correct legal reasoning follows, the 70B prefers the word "punishable" for about a third of private settings and unlisted groups.

What's next

  • More statutes with statute-twins new.
  • Training targets for the next Apertus: the closed list, the in-force dates, the law as written.

Built with Claude Code agents, declared in the repository. Research data, not legal advice.

Built With

Share this project:

Updates

Submission history