VeritasAI

Every claim deserves a trial

Ask any single LLM whether social media causes teen depression and you get the same shrug: "correlation isn't causation, more research is needed." It isn't wrong. It's just useless — you can't do anything with it.

The legal system solved this problem a long time ago. The way to find truth isn't to ask one person what they think. It's to force the strongest possible case for each side, then let a neutral party decide. We wanted to know whether that structure would work if the advocates were LLMs.

It does — and the reason is more interesting than we expected. A model given a role and a duty stops hedging. Hedging is what a model does when it has no stake. Give it a side to argue and it goes and finds the evidence.

Live app: https://veritasai-xi.vercel.app

What it does

VeritasAI runs a claim through a seven-stage adversarial pipeline in about eight seconds:

#	Stage	Model	Family
0	Court Researcher — retrieves live evidence before anyone argues	no model	CrossRef + Wikipedia + DuckDuckGo
1	Claim Decomposer — splits the claim into checkable sub-claims	`llama-3.1-8b-instant`	Llama
2	The Prosecution — strongest case the claim is false	`llama-3.3-70b-versatile`	Llama
3	Expert Witness — neutral read on what the evidence says	`gpt-oss-120b`	GPT-OSS
4	The Defense — strongest case the claim is true	`gemma-4-31b`	Gemma
5	The Judge — weighs all three briefs and rules	`zai-glm-4.7`	GLM
6	Citation Clerk — verifies every cited source against CrossRef	no model	CrossRef

The verdict is a label (TRUE → FALSE), a 0–100 truth score, a confidence rating, the reasoning, the strongest argument each way, specific nuances, and a citation audit that flags sources that don't exist.

Five roles, five different model families, two providers. That matters: the Prosecution (Llama) and the Defense (Gemma) come from different pretraining distributions on purpose. The whole premise collapses if both briefs are one model arguing with itself.

How we built it

Python 3.14 for the reference implementation (CLI + browser extension), then a TypeScript port on Next.js 15 + Tailwind v4, deployed to Vercel. All five models run on the free tiers of Cerebras and Groq. The web UI streams progress over Server-Sent Events, so you watch each party file its brief live.

Everything that can overlap does: retrieval runs alongside decomposition, the three briefs run concurrently, and the citation audit runs alongside the ruling.

The prompt engineering is the actual project

Persona before task. Every node opens by fixing an identity and a duty, not a question. This is the single biggest lever on hedging.
Adversarial licensing via epistemics, not orders. The Defense is told why its job is legitimate — "you cannot know a claim is false until someone competent has tried to defend it" — rather than commanded to comply.
Mandatory concessions. Both advocates must concede points. This is what lets a model argue hard without feeling it's lying, and it hands the Judge a pre-marked map of where the sides already agree.
Numeric confidence before prose. Each advocate commits to a 0–100 score, giving the Judge a quantitative prior before it reads any rhetoric.
Explicit anti-hedging on the bench. The Judge is told that retreating to MIXED when the record supports a real verdict is "a failure of nerve", and is given score bands per verdict label.

Challenges we ran into

Getting a model to argue for something it "knows" is false. This was the core problem. Direct instruction fails — models break character and emit a disclaimer instead of a brief. What worked was explaining the epistemic role of a defense and pairing it with a mandatory concessions field and an honest confidence score. That decouples argue hard from report honestly, and once a model has somewhere to put its actual belief, it stops fighting the task.

JSON from open-weight models. Far harder than expected, and all of these were observed live rather than anticipated:

Reasoning models emitting <think> blocks, sometimes truncated mid-thought
llama-3.3-70b-versatile reliably emitting unquoted string values — "headline": The claim is false, — with the rest of the object perfect
Responses truncated mid-string when a model ran out of budget
Schema drift: arguments for evidence, scores as 72 / "72" / "72/100" / 0.72

We built a layered salvage path — strip, close, quote, re-ask — and only fall back to a repair round trip when all of it fails.

A repair loop that made things worse. Our first repair implementation replaced the conversation history with the repair instruction. The Judge, having lost the briefs it was supposed to rule on, solemnly reported that no arguments had been provided — and returned a confident-looking MIXED 50/100. It looked like a model failure. It was our bug.

Model IDs that didn't exist. We hardcoded a roster from memory and every single ID was wrong. The roster is now verified against each provider's live `/v1/model