Inspiration
A trust report that says "Supported by 6 sources" sounds decisive. But what if 3 of those sources were published before the event they describe, and two are the same article listed twice? Recent research shows the same problem with AI judges: a panel of 9 LLM judges from 7 model families carries about the information of 2 independent votes, because their errors are correlated (Kohli, 2026; Kim et al., ICML 2025). The Condorcet jury theorem only works when votes are independent. We wanted a trust agent that asks one question before anyone acts: how much independent evidence is actually behind this?
What it does
Condorcet Agent reads an Ezio trust report and decides what happens next: APPROVE, REWRITE, PULL_SOURCE, ESCALATE or BLOCK, for a specific person and use (e.g. an analyst putting an AI answer into a client note).
- Audits the evidence (deterministic code). For every claim it drops sources published before the period the claim covers, and groups the rest by shared origin (same outlet, near-identical titles, same story within a day).
- Checks the verifier. It flags Ezio verdicts that contradict their own evidence.
- Runs a judge panel of 9 models from 9 different families (OpenAI, Anthropic, Google, DeepSeek, Qwen, Mistral, xAI, Zhipu, Moonshot) via TokenRouter.
- Asks one targeted question to the accountable human; the answer becomes evidence and the decision re-runs (before → after).
- Safety floor in code: a contradicted claim or a critical finding can never be approved, whatever the judges say.
How we built it
Python, httpx, TokenRouter (OpenAI-compatible), a static HTML UI. We measure a panel's effective number of independent judges with Kish's formula:
$$ n_{\text{eff}} = \frac{n}{1 + (n-1)\,\bar{\rho}} $$
where $\bar{\rho}$ is the average pairwise error correlation. With $n = 9$ and $\bar{\rho} \approx 0.44$, $n_{\text{eff}} \approx 2$.
Inspired by the random subspace method behind random forests (Ho, 1998; Breiman, 2001), we also built an evidence-subspace panel: each evidence piece is shown to $r = \lceil 0.4\,n \rceil = 4$ of the 9 judges, assigned round-robin with a seeded shuffle so every piece reaches exactly $r$ judges. Every decision also carries a simple-rule baseline and a flag for whether the panel added anything beyond it.
What we found (on the four Ezio sample reports)
- EV market: the 2024 sales claim lists 6 sources, but only 3 could confirm it (3 were published before 2024 ended).
- AGI timelines: a claim lists 4 sources → 1 usable origin (the same WSJ article twice, plus 2 published too early).
- Nuclear: the same CNN article is listed twice; "6 official sources" include 4 Department of Energy pages (6 → 3 origins).
- Work from home: the only supported claim cites the same Chicago Booth Review article 3 times.
- Verifier error: Ezio marks the EV Geely figure "contradicted" while quoting the same 1.49M figure. None of the 8 real AI judges noticed; our code check did.
What we learned
- We tested our own hypothesis, and it failed on this case. With real models on the EV report, judges who all saw the full evidence agreed 7 of 8 (REWRITE). Splitting the evidence lowered agreement (4 vs 2): hiding evidence created disagreement rather than revealing it. We report that, label such splits as an information gap instead of uncertainty, and use the split only to name the evidence a reviewer must not miss. It's one report, so it's evidence, not proof.
- Deterministic checks caught what every AI judge missed. Agreement among models, or among sources, is not the same as independent confirmation.
Challenges
- Pivoting under time pressure: our first design was a research experiment that fit no official track; we rebuilt it as a trust agent inside the window.
- No export from Ezio: we transcribed all four sample reports from screenshots and disclosed this in every file.
- Rate limits: TokenRouter allows 15 requests per minute; we added throttling, retries and an explicit rule that a failed judge is excluded, never counted as a vote.
- Honest evaluation: an external review pointed out that our panel's "flip evidence" matched what a simple rule finds. We added the rule baseline to every decision and ran the real-model control to check it ourselves.
