• 

  EpiTrace tests whether AI confuses repeated reports with independent evidence, and whether it recognizes uncertainty about sources.

• 

  39 synthetic scenarios, six evidence conditions, and 234 tests per model reveal how repetition and source relationships affect AI judgments.

• 

  Across 468 real Apertus evaluations, both models handled known copies well but failed all 78 unknown-provenance observations.

• 

  An open-source Python benchmark with frozen datasets, real model responses, automated scoring, 25 tests, and offline reproducibility.

EpiTrace — Epistemic Integrity Under Repetition

Five newspapers do not necessarily mean five independent sources.

Inspiration

In my research on rumours circulating through nineteenth-century newspapers, I encountered a recurring problem: the same claim could appear in multiple publications without gaining any genuinely new evidence. Repetition can make information seem more credible, even when every report originates from the same source.

Today, AI systems increasingly summarize news, research documents, and intelligence reports. But can they distinguish repeated information from independent corroboration? And what happens when the relationships between sources are unknown?

I created EpiTrace to investigate these questions using Apertus, Switzerland's open language model.

What it does

EpiTrace is an open-source benchmark for evaluating how language models handle evidence repetition, source dependence, and uncertainty.

It tests whether a model can distinguish three important situations:

• YES: Evidence comes from independently identifiable sources.
• NO: The reports are known to share an origin.
• UNKNOWN: The available information does not establish their relationship.

Starting from 39 original fictional newspaper-style scenarios, EpiTrace generates six controlled conditions per scenario: a baseline report, three identical copies, five identical copies, five explicitly linked newspaper reports, five reports with untraceable origins, and reports containing independent firsthand evidence.

This produces 234 evaluation cases per model.

How I built it

I implemented the benchmark in Python with deterministic dataset generation, structured prompts, JSONL response logging, automated scoring, and paired statistical comparisons.

I tested Apertus 1.5 8B and 70B using the Public AI API, collecting 468 genuine model responses.

The evaluation measures epistemic-state agreement, stability under repetition, recognition of source independence, and changes in estimated claim support.

For reproducibility, the repository includes the synthetic dataset, original model outputs, checksums, 25 offline tests, Docker configuration, and an offline replay mechanism that reconstructs the analysis without API credentials or additional inference costs.

Challenges and what I learned

My initial expectation was that repeating the same report might cause the models to mistake repetition for stronger evidence.

The results challenged that expectation.

Both Apertus models performed well when source relationships were explicitly documented:

• Apertus 8B correctly recognized all 117 known-copy conditions.
• Apertus 70B correctly recognized 115 of 117.
• Both recognized all 39 independent-evidence conditions.

However, neither model correctly selected UNKNOWN in any of the 39 unknown-provenance cases. Instead, both consistently answered NO, treating insufficient information about source origins as evidence that the reports were dependent.

This distinction is fundamental: not knowing whether sources are independent is different from knowing they are not independent.

Other challenges included obtaining accessible model inference, controlling experimental conditions, defining defensible reference labels, and making the results reproducible. I ultimately used Public AI rather than claiming an unverified Swiss-hosted deployment.

Importantly, the benchmark uses author-labeled synthetic examples, not independently adjudicated historical records. Its findings are diagnostic and should not be generalized to all real-world applications without further testing.

What's next

I want to extend EpiTrace with historical newspaper collections, paraphrased reports, OCR errors, independently annotated source relationships, and more diverse ambiguous-provenance cases.

The longer-term goal is to support AI systems used in journalism, historical research, fact-checking, and information management.

EpiTrace is not simply about detecting misinformation. It is about teaching AI evaluation to respect the difference between repetition, corroboration, and uncertainty.

Built With

• 1.5
• actions
• ai
• apertus
• api
• docker
• evaluation
• github
• jsonl
• llm
• make
• matplotlib
• natural-language-processing
• public
• python
• reportlab
• reproducible
• requests
• rest
• sha-256
• unittest