Inspiration
Swiss voting booklets are published in German, French and Italian, and claims about them travel in all three languages. We wanted a system that does more than give a label. It should also point to the exact passage in the original booklet, in the original language, so anyone can check the answer. We also wanted to see how far the open Apertus models (8B and 70B) could go when the surrounding code does the careful bookkeeping.
What it does
The input is a claim in German, French or Italian, together with either a full voting booklet PDF and the vote name (Task A) or a short reference text (Task B). The system decides whether the claim follows from the source (0, entailment), lacks support (1, neutral) or contradicts it (2, contradiction). For booklet cases it also returns verbatim evidence in the original language with physical page numbers.
How we built it
The model never writes quotations. A script extracts each PDF page by page with pypdf, picks the pages that belong to the vote, and cuts them into source spans with IDs. The model returns only a label and up to five span IDs through a single native submit tool call. The script then rebuilds the quotation and the page number from the original text.
- Page selection. Federal booklets print ordinal running headers such as Erste Vorlage / Premier objet / Primo oggetto, and we assign pages to a vote using these headers.
- One call per case. Task A (booklet) is answered by Apertus 70B, Task B (reference text) by Apertus 8B.
- Paragraph context. For Task A we split the vote's pages into 200–700 character paragraphs and rank them by fusing BM25 over an 8B translation of the claim with a small local multilingual embedding. Only the top 8 paragraphs and their neighbours go to the model. If it cannot cite evidence for a non-neutral answer, we re-run that case once on the full vote pages.
Challenges we ran into
- Missing pages. Our earlier page selector left out the pages with the gold evidence for 14 of 39 checked questions (36%), mostly the summary spread.
- Structured output. Asked for plain-text JSON, the 8B model's Task A $\text{F1}$ dropped to 0.361, so native tool calls became mandatory. We also check every
submit: we reject unknown IDs, more than five IDs, and non-neutral answers without evidence, and ask the model to try once more. - Cross-language retrieval. Word matching misses many claim/booklet pairs in different languages. Paragraphs alone also lowered Task A $\text{F1}$, because contradictions often could not be quoted. Adding the full-page re-run fixed this.
- Messy PDFs. For some questions the gold evidence does not match pypdf's extracted text at all.
Accomplishments that we're proud of
- Simpler and cheaper. We first ran a 70B parent that consulted an 8B child. In all 420 recorded cases, the 70B's final label was identical to the 8B's suggestion. The parent's two calls used 63% of the tokens and 65% of the waiting time while adding no judgment, so we moved to one call per case.
- Better with less context. Averaged over our three 60-question sets, the paragraph setup compared with sending full vote pages moved Task A $\text{F1}$ from 0.881 to 0.903 and Hit@5 from 0.746 to 0.812. Tokens per case fell from 10,939 to 7,629 (about 30% fewer). Time rose from 3.7 s to 4.0 s because of the translation and re-run calls.
The differences are small, and each setup was run only once per set. Results can shift by a few cases between runs even at temperature 0. Most of the $\text{F1}$ gain came from one set, and the other two barely changed. The token reduction is the clearer result. All numbers come from the public training data, and we used all three sets during development.
What we learned
- Look at what the extra model is actually doing. Logging the 70B and 8B answers side by side showed that the second model added cost and no judgment.
- Having the evidence in the input is not enough. After fixing page selection, the model still often cited other pages, so we added evidence completion around its citations.
- "No evidence, then neutral" works as a rule. Most cases that fell back to neutral were neutral in the gold labels too.
What's next for OST
- Run the final setup several times on each set to measure run-to-run variation.
- Check whether region-level evidence now matches the cases where pypdf's text did not match the gold evidence, and compare other PDF extraction libraries for the rest.
- See how the system does on the held-out data, which we have not seen.
Log in or sign up for Devpost to join the conversation.