SwissGov-RSD DiffAlign

Inspiration

Swiss government websites publish every page in German, French, Italian, and English, and all four versions are meant to carry the exact same information. In practice, translation inaccuracies and unsynchronized updates cause real semantic divergences between texts. The SwissGov-RSD benchmark evaluates whether a model can detect these divergences token by token.

Prompted LLMs score poorly on this task, often close to zero. That outcome aligns with findings from a bachelor thesis on cross-lingual alignment (MEXA): multilingual decoders construct a shared representation space in their middle layers, and Apertus has an anisotropic geometry where raw cosine similarity obscures semantic signals. The model possessed the necessary information; prompting was just the wrong mechanism to access it.

What it does

Given two related documents in different languages (English paired with German, French, or Italian), the system assigns each whitespace-separated token a score reflecting how much its meaning diverges from the aligned text in the counterpart document. This flags additions, omissions, and mistranslations. It uses swiss-ai/Apertus-v1.5-8B directly without prompting or fine-tuning.

How we built it

We treat Apertus as an encoder rather than a generative chatbot, using an approach we call DiffAlign-Apertus:

1. Encode each document once with Apertus and extract hidden states from a chosen intermediate layer.
2. Score each token $a_i$ in document A as the complement of its strongest cosine similarity to any token in document B:

$$ \text{diff}(a_i) = 1 - \max_j \cos\big( h(a_i),\, h(b_j) \big) $$

1. Mean-center the token representations prior to scoring, which removes the language-identity offset dominating Apertus's vector space:

$$ \tilde{h}(a_i) = h(a_i) - \frac{1}{n}\sum_{k=1}^{n} h(a_k) $$

1. Map subword scores back to whitespace words using tokenizer character offsets so predictions line up with benchmark annotations.

Because the approach is completely unsupervised, we never touched the held-out test split during development. We configured all settings on the dev split and froze them there. Experiments ran in native bfloat16 on a single NVIDIA H100 hosted by LRZ AI Systems, alongside an interactive dashboard to visualize token diffs, layer sweeps, and runtime efficiency.

Challenges we ran into

• Apertus-v1.5 is an Omni multimodal checkpoint rather than a standard causal model. Initial runs returned NaNs because hidden states came from the outer wrapper; we had to register the architecture properly to access the inner language model.
• Subword-to-word misalignment degrades token-level Spearman correlation quickly. Aggregating subwords back to words via exact character offsets turned out to be strictly necessary.
• Cluster workflow: managing Slurm job queues, fitting 8B parameters into memory in bfloat16, and completing H100 runs before the deadline.

Accomplishments that we're proud of

• Reached 0.255 macro Spearman correlation on the SwissGov-RSD dev split without training, well above published prompting baselines (around 0.02 to 0.04) and prior unsupervised DiffAlign numbers (0.155).
• Mean-centering roughly doubled macro Spearman at the final layer (0.078 to 0.154), demonstrating the practical effect of correcting vector anisotropy.
• Achieved high throughput: roughly 0.15 seconds per document pair on ~16.5 GB VRAM with a single forward pass, completely eliminating the JSON and token-count formatting failures common to prompting.

What we learned

• The main bottleneck in Apertus was representation geometry rather than comprehension: centering embeddings surfaced the latent signal.
• Cross-lingual alignment peaks at intermediate depth (layer 20 of 32, or 62.5% depth), matching the hypothesis that shared representations develop midway through the decoder.
• Extracting hidden states directly from decoder LLMs is a fast, robust alternative to prompt engineering for fine-grained token-level tasks.

What's next for SwissGov-RSD DiffAlign

• Evaluate the frozen configuration (--layer 20 --center) on the held-out test split for the official leaderboard submission.
• Run a finer sweep across layers 17 to 22 and test blending activations from complementary layers to improve dev performance.
• Investigate why scores are consistently lower on the English-French language pair.
• Evaluate a 4-bit quantized 70B model to examine the trade-off between latency and accuracy.

Built With

• apertus
• conda
• cuda
• huggingface
• pytorch
• scipy
• slurm
• sovereignty
• swiss-ai
• transformers
• typescript