FAULTLINE — Evidence-Driven Red-Teaming of Apertus 1.5

Inspiration

An open and multilingual language model should remain trustworthy across languages, cultures, and conversational contexts. Yet evaluating a model through isolated prompts may overlook inconsistencies that emerge when the same question is translated, reframed, or presented through multiple conversational turns.

FAULTLINE was conceived to investigate these blind spots through systematic, evidence-driven red-teaming of Apertus 1.5.

Our goal is not simply to collect surprising responses. We want to identify meaningful model failures, understand the conditions that trigger them, and provide reproducible evidence that can help improve open and trustworthy AI.

What We Are Investigating

FAULTLINE focuses on three complementary areas:

1. Cross-Lingual Factual Integrity

Testing whether semantically equivalent prompts across different languages produce inconsistent factual claims, misleading historical explanations, or unjustified differences in confidence.

2. Cultural Representation and Bias

Investigating whether linguistic framing, cultural context, or demographic references introduce stereotyping, unfair assumptions, or misrepresentation of cultural values.

3. Context-Dependent Safety and Reliability

Examining whether paraphrasing, conversational history, and multi-turn interactions expose failures that are not observable in isolated evaluations.

These categories are investigation targets, not claims that such vulnerabilities have already been discovered.

Methodology

Our planned evaluation follows an evidence-first workflow:

1. Threat and failure modeling: Define the expected behavior, risk category, and measurable failure condition for each investigation.
2. Controlled test generation: Prepare comparable prompt variants across languages, wording, and conversational contexts.
3. Model evaluation: Test Apertus 1.5 under documented model versions and inference configurations.
4. Failure validation: Reproduce suspected issues, examine alternative explanations, and distinguish genuine failures from ambiguous or unsupported judgments.
5. Evidence packaging: Record prompts, outputs, relevant settings, reproduction steps, limitations, and proposed mitigations.

Rather than prioritizing attack volume, FAULTLINE emphasizes the quality, distinctiveness, and reproducibility of the reported findings.

Technical Challenges

Several methodological challenges are central to this work.

Translations may not preserve identical meanings, making apparent cross-lingual inconsistencies difficult to interpret. Factual disagreements must be evaluated against reliable reference material, and isolated model responses may not represent stable behavior.

To address these issues, the evaluation will use explicit success criteria, controlled comparisons, repeated testing where feasible, and human review of ambiguous cases.

Intended Deliverables

FAULTLINE aims to produce:

• A curated collection of independently verified red-team findings.
• Reproducible prompt-and-response evidence for each finding.
• A documented evaluation methodology and reusable testing artifacts.
• Technical explanations of observed failure mechanisms.
• Practical recommendations for model evaluation and improvement.

Project Status

This description presents the evaluation design. Confirmed findings, quantitative results, implementation details, and lessons learned will be added after experiments are completed and validated.

The ultimate objective is to turn model failures into actionable evidence that advances the reliability of Apertus and contributes to the broader open-source AI safety community.

Built With

• apertus-1.5
• github
• hugging-face-transformers
• pandas
• pytest
• python
• pytorch