AI Debate Verifier

This is the tech stack that has been used
This is the code pasting area
This explains the proper flow of the project

Inspiration

Every AI code-review tool has the same flaw: you ask one LLM "is this code safe?" and it confidently agrees with itself. Hallucinations go unchecked and missed vulnerabilities stay missed, because nobody is motivated to argue back. Courtrooms solved this centuries ago — you don't ask one lawyer for the truth, you make two argue and let a judge decide. We wanted to see if the same adversarial structure could make LLM code verification actually trustworthy.

What it does

AI Debate Verifier puts your code on trial. Paste a Python snippet and the pipeline runs:

Static Analysis → Claim Generator → Defender vs Challenger → Judge → Fixer (Bandit+Semgrep) (testable claims) (N rounds, parallel) (verdict) (patch)

Static analysis first — Bandit (https://bandit.readthedocs.io/) and Semgrep (https://semgrep.dev/) scan the code, so the debate is grounded in real evidence, not vibes.
Claims are generated — an LLM distills findings into concrete, testable claims like "This code is safe from SQL injection."
The debate — for each claim, a Defender argues the code is safe (citing exact lines) while a Challenger attacks with concrete exploit paths and sample payloads, over multiple rounds. Claims are debated in parallel.
The verdict — an impartial Judge scores both sides on factual accuracy and logical support, picks a winner, and classifies why the loser failed:

{ "winner": "challenger", "scores": { "defender": {"factual_accuracy": 4, "logical_support": 3}, "challenger": {"factual_accuracy": 9, "logical_support": 9} }, "failure_type": "missed_security_gap", "explanation": "The defender never addressed the unsanitized f-string reaching cursor.execute()..." }

The fix — a Fixer agent produces one c patch for exactly what the Challengerproved broken, shown as a diff.

The entire pipeline streams live to the browser — stages light up, arguments land turn by turn, and verdicts arrive in real time.

How we built it

Orchestration: LangGraph (https://www.langchain.com/langgraph) StateGraph with four nodes over a shared typed state — every node pushes progress onto LangGraph's custom stream.

graph = StateGraph(DebateState) graph.add_edge("baseline", "claim_generator") graph.add_edge("claim_generator", "debate_claims") graph.add_edge("debate_claims", "fixer")

Backend: FastAPI forwards those events as Server-Sent Events; a sync endpoint reuses the exact same graph.
LLMs: Groq (https://groq.com/) running openai/gpt-oss-120b, with per-role tuning — reasoning effort dialed low for cheap tasks and high for the Fixer, plus Groq's JSON mode for reliably structured verdicts and patches.
Frontend: Next.js 16 + React 19, with a hand-rolled SSE client (fetch + ReadableStream — EventSource can't POST) feeding a useReducer state machine that drives the live pipeline visualization. ## Challenges we ran into
Rate limits shaped everything. Groq's free tier allows ~8k tokens/min and one run makes 10–15 LLM calls. We budgeted tokens like a scarce resource: one-line-per-finding prompt summaries, capped claims and parallelism, max_tokens limits, and aggressive retry/backoff so runs degrade gracefully instead of dying on a 429.
Streaming from worker threads. LangGraph's stream writer lives in Python contextvars, which don't propagate to ThreadPoolExecutor workers — our parallel debates silently emitted nothing until we ran each worker inside contextvars.copy_context().
LLMs are bad at emitting JSON. Multi-line code inside JSON strings broke constantly. The fix was layered: Groq's server-side JSON mode, a tolerant parser that finds the largest valid JSON span in any response, and semantic fallbacks — an unparseable judge verdict defaults to whichever side the static analyzers support.
Never let one failure kill a run. A broken scanner, a rate-limited debate, a failed fixer — each degrades into a labeled partial result, never a 500. ## Accomplishments that we're proud of
A judge that doesn't just pick a winner but classifies the failure mode — hallucination, unsupported_logic, or missed_security_gap — turning "the model was wrong" into measurable categories.
Full real-time streaming of a multi-agent pipeline: every stage, turn, and verdict renders live.
Evidence-grounded debate — agents must cite exact lines against real scanner findings, and a scanner failure is explicitly surfaced so it can never masquerade as a clean scan.
The whole thing survives the free tier: retries, fallbacks, and partial transcripts mean a run always finishes with something honest. ## What we learned
Adversarial structure genuinely changes LLM output — a defender forced to answer a specific exploit path argues very differently from a model asked "is this safe?"
LangGraph's value beyond orchestration: one graph serves both streaming and sync endpoints, and custom stream mode makes agent UIs almost free.
Production LLM engineering is mostly about failure: token budgets, rate limits, malformed output — and designing fallbacks that fail toward the evidence, not toward a coin flip.
SSE + a reducer state machine is a clean, WebSocket-free recipe for live multi-agent frontends. ## What's next for AI Debate Verifier
More languages — the debate engine is language-agnostic; only the scanners are Python-focused today.
Dynamic evidence — sandbox the Challenger's proposed exploit and actually run it, making verdicts empirical.
Verify the fix — feed the Fixer's patch back through the pipeline until the Challenger can no longer win.
Persistence & history — LangGraph checkpointing for resumable runs and failure-mode analytics across a codebase.
CI integration — a GitHub Action that puts every pull request on trial.

Notes: I used text,json, and ```python blocks for the pipeline diagram, verdict example, and graph wiring — those render nicely on Devpost. No LaTeX, since there's no math here to show. If you have a public GitHub repo link, tell me and I'll drop it in (e.g., in "How we built it").