CRUX Adversarial Claim Verification

Home page

Inspiration

Standard Large Language Models (LLMs) are often "confidently wrong" when asked to verify facts, occasionally fabricating evidence or falling prey to cognitive biases. In a single-prompt setup, an LLM behaves like a judge who only hears one side of a story before delivering a verdict.

CRUX was inspired by the legal system and classical debate structures. Instead of asking a single AI to decide if a claim is true, we force two AI agents—an Advocate (defense) and a Destructor (prosecution)—to argue opposite sides in a structured debate. By isolating the single factual or logical point on which their disagreement actually hinges (the Crux), and submitting it to an independent Judge agent, we get a significantly more accurate, calibrated, and nuanced verdict.

What it does

CRUX is an automated, real-time agentic debate pipeline that evaluates factual and statistical claims:

The Advocate: Builds the strongest possible case arguing that the claim is TRUE. The Destructor: Attacks the Advocate's weakest arguments and presents a case that the claim is FALSE or MISLEADING. Crux Extraction: Both agents are forced to name the exact point where their disagreement lies (the Crux) in under 50 words. The Judge: Acts as a neutral arbitrator. It reviews the arguments, classifies the crux's evidence type, and outputs a final calibrated verdict (TRUE, FALSE, MISLEADING, or AMBIGUOUS) and a confidence score (0-100%). Calibrated Capping: If the Judge determines the crux is a definitional dispute or subjective assumption (rather than a verifiable physical fact), the system prompt automatically caps the confidence score below 60% to prevent overconfidence. Real-time SSE Dashboard: Users input claims in a sandbox and watch the debate stream stage-by-stage in real-time, side-by-side with a single-agent baseline comparison.

How we built it

Backend Architecture: Built with FastAPI (Python) and SQLite for storing claims and historical debates. Agent Orchestration: Structured using asynchronous workflows in Python. We used Asyncio to run independent tasks (like the single-agent baseline and dual crux extractions) concurrently. LLM Integration: Integrated via OpenRouter pointing to google/gemini-2.5-flash (and Qwen models), utilizing custom exponential backoff retry wrappers to handle rate limits. Real-time Streaming: Implemented Server-Sent Events (SSE) to stream intermediate arguments to the frontend instantly as they generate. Frontend Web UI: An elegant, high-fidelity dashboard built using Vanilla HTML/CSS/JS featuring dark mode, glassmorphism, responsive tables, and custom transitions for the streaming debate states.

Challenges we ran into

Sequential Latency: Spawning multiple agents sequentially took up to 30 seconds per claim. We solved this by parallelizing independent agent tasks (e.g., launching the baseline concurrently with the debate, and running crux extraction in parallel) and enforcing strict token limits, cutting latency by 2.5x. Pedantry Bias: Initially, the Destructor agent would invent minor semantic exceptions (e.g., "Venus's temperature depends on average vs. peak definitions"), causing the Judge to over-classify standard facts as MISLEADING. We resolved this by embedding strict definitions of TRUE and MISLEADING in the Judge's prompt, instructing it not to penalize claims for pedantic nuances if they are widely accepted as fact. API Rate Limiting: Sending multiple rapid requests concurrently caused 429 errors. We implemented a robust retry mechanism with exponential backoff and capped response token usage.

Accomplishments that we're proud of

100% Accuracy on Test Subset: After prompt optimization, the pipeline achieved 100% accuracy on standard factual claims, successfully bypassing the pedantic biases that plagued the initial run. High-Concurrency Performance: Optimizing the async architecture reduced average latency from 30+ seconds down to ~12 seconds per claim. Calibrated Confidence Capping: Successfully forcing the LLM to lower its confidence when dealing with subjective or ambiguous parameters.

What we learned

Structured disagreement is superior: Forcing AI agents to disagree and find their point of tension highlights nuances that single prompts completely miss. Conciseness boosts speed: Imposing strict word counts (under 120 words for arguments) not only reduced execution latency but also improved reasoning clarity by eliminating LLM fluff. Calibrated Certainty: A system that knows why it is uncertain (identifying definitional vs. factual disputes) is much more trustworthy than one that presents all verdicts with equal confidence.

What's next for CRUX Adversarial Claim Verification

Web-Search Integration (RAG): Integrating Google Search or Bing API so that the Advocate and Destructor can back their arguments with real-time web citations. Multi-Turn Debates: Allowing the Advocate to respond to the Destructor's counter-arguments before extracting the crux. Hierarchical Judges: Spawning a panel of three Judges and using a voting consensus model to verify highly controversial claims.