Inspiration
LLMs are getting frighteningly good at sounding right. They answer multi-hop questions and confidently cite their sources — but is the source real? Does it actually support the claim? We kept seeing models that get the right answer while pointing to evidence that's invented or irrelevant. Standard benchmarks score the answer and stop there. We wanted one that scores the evidence.
What it does
Citation Trap puts an LLM in the role of a research analyst. It answers a multi-hop question over a Wikipedia-derived corpus and must cite a passage for every claim it makes. We score two things — citation faithfulness (do the cited passages actually support the claim?) and answer correctness — and cross them into a 2×2 (shown in the cover image):
- ideal — answer right, citations trustworthy
- correct_but_fabricated — answer right, citations invented/misattributed ← the headline
- honest_wrong — answer wrong, citations trustworthy
- worst — answer wrong, citations fabricated
The cell that matters is correct_but_fabricated — right answer, invented sources — the failure mode a normal benchmark can't see.
Real example (q53): asked which war the Livesey Hall memorial commemorates, the agent answers "World War II" — correct — then adds "World War II had over 60 million casualties" and cites a passage id that doesn't exist in the corpus. Right answer, fabricated source.
How we built it
- Data: 500 HotpotQA (distractor) questions, ~5,000 passages, built through quality gates (every answer verified to actually appear in a gold passage, yes/no answers capped, dedup). All ~5,000 passages live in one shared BM25 index, so the right support competes with thousands of look-alikes.
- Scoring, two backends: a deterministic gold-set proxy (offline) and an LLM entailment judge that decides whether the cited passage genuinely supports the claim — not just whether it's the blessed gold passage. Passage ids are opaque hashes, so an agent must actually retrieve a passage to cite it.
- System under test: any OpenAI-compatible model via a small registry; we ran DeepSeek-chat and -reasoner locally.
- UI: a single-page trace viewer plus a model leaderboard, and a local server for in-browser live runs.
- Platform: wrapped the core onto SWECC's mesocosm / BenchAnything four-endpoint protocol and submitted it as a live developer environment.
Results
- Local (entailment-judged, 500 questions): DeepSeek-chat 93.3% faithful vs DeepSeek-reasoner 92.7%. A finding we're proud of: a 100-question run suggested the reasoner cited less faithfully — scaling to 500 showed that gap was small-sample noise. A citation benchmark only earns trust at scale.
- On the mesocosm platform (gold-proxy scored): Claude Sonnet 4.6 (42.5%) beats GPT-4o (26.7%) — both genuinely search and cite.
Challenges we ran into
Running on a live platform surfaced bugs we'd never have seen locally:
- A thorough agent (Claude) searched our 5,000-passage corpus until it ran out of steps and never submitted, scoring 0 — so we added search caps that force a decision.
- We isolated a platform-side issue: all 7 Gemini models emit an empty action and score 0, while Claude and GPT-4o work fine on the same environment — a structured-output gap we filed as a bug report.
- Plus invalid model names, intermittent sandbox 502s, and a stricter-than- documented manifest schema.
What we learned
- Measuring faithfulness is harder than measuring correctness, full of subtle false positives — e.g. don't penalize a model for not citing a pure deduction ("1882 is earlier than 1954"). We designed the rubric to separate "lied about a source" from "drew a conclusion from cited facts."
- Building platform-agnostic first paid off: integrating with mesocosm was a thin shim, and the same core ran locally and on the platform.
What's next
The entailment judge on the platform, more models, and adversarial distractors to push the trap harder.
Log in or sign up for Devpost to join the conversation.