CodeTribunal

Your code is guilty until proven innocent. 5 adversarial AI agents — prosecutor, defense, judge & more — debate every vulnerability in a live courtroom trial powered by Qwen.

Comment

Inspiration

Most AI code review tools work the same way: one model reads your code, lists some findings, and trusts itself completely. There's no second opinion, no pushback, no "wait, is that actually true?"

That bothered me. Real code review doesn't work that way — a senior engineer challenges a junior's flagged "bug," asks for evidence, and sometimes the finding gets walked back. That adversarial back-and-forth is where the actual signal comes from. A single LLM pass can't replicate it, because it never has to defend its own claims against someone trying to tear them down.

So I asked: what if code review was a trial? What if a security finding had to survive cross-examination before it counted against your score?

That's CodeTribunal — a courtroom where your code is the defendant, and five AI agents argue its fate.

What it does

You submit a code file. Five agents go to work:

  • LEDGER (Clerk) — parses the code via AST, files the case: line counts, functions, imports, structure.
  • AEGIS (Prosecutor) — hunts for vulnerabilities using Bandit, builds accusations with confidence scores and exact line citations.
  • AXIOM (Defense) — runs validation-pattern detection and AST proof to counter false positives, conceding only when the evidence is real.
  • METRIC (Expert Witness) — brings complexity and performance data (Radon) as objective testimony.
  • ARBITER (Judge) — detects which findings actually conflict, runs cross-examination rounds only on disputed clusters, and renders a final verdict.

The trial isn't theater. If AEGIS claims subprocess usage is a command-injection risk and AXIOM proves via AST analysis that subprocess is never actually called, ARBITER dismisses that finding — and it's excluded from the security score entirely. If AEGIS can't produce a real counter-argument, the system forces a concession (confidence must drop to ≤0.2). Findings that survive cross-examination are CONFIRMED and penalized at full weight; ones that get refuted are DISMISSED and removed from scoring; genuinely ambiguous ones are DISPUTED and penalized at half-weight. The score is computed after the debate, not guessed by an LLM before or after the fact.

The whole thing is presented as a visual novel courtroom — custom-illustrated character sprites, typewriter dialogue, click-to-advance pacing — because watching five agents debate your code's fate should feel like watching a trial, not reading a log file.

How we built it

Backend (FastAPI + Python): Five agents, each calling Qwen Cloud models tiered by task complexity — qwen-max for ARBITER and AEGIS (heavy adversarial reasoning), qwen-plus for AXIOM and METRIC (counter-argument and data analysis), qwen-turbo for LEDGER (mechanical AST parsing). The orchestration runs investigation in parallel (asyncio.gather), then does deterministic conflict detection — comparing line ranges with ±3 line tolerance, zero LLM calls — to figure out which findings actually need a debate. Only contested clusters go to cross-examination; uncontested findings skip straight to verdict. This keeps token usage proportional to actual disagreement, not blanket re-litigation of everything.

Real-time proceedings stream to the frontend over WebSocket, backed by session state in Neon Postgres so reconnects and multi-worker deployments stay consistent.

Frontend (Next.js + Framer Motion): A full-screen visual novel interface. Each agent has a custom AI-illustrated sprite (multiple poses — neutral, accusing, conceding, ruling), a themed dialogue box with backdrop blur, and a typewriter effect with click-to-skip and adjustable playback speed. ARBITER takes over the full screen for the final verdict, gavel and all.

The scoring pipeline was the part we iterated on hardest. Early versions had the LLM both write the verdict and report its own numeric score in the same response — which meant the model's self-reported number could (and did) contradict the deterministic score computed from actual finding statuses. We fixed this by stripping all scoring authority from the LLM's free text: it writes per-finding rulings only (CONFIRMED/DISMISSED/DISPUTED), the system parses those statuses back onto the finding objects, then computes the rubric score from that — with dismissed findings excluded and disputed findings at half-weight. The LLM never sees or reports a number; the system appends the real one after the fact.

Challenges we ran into

  • The "kangaroo court" problem: early versions had AXIOM win an argument in the dialogue but the score didn't change, because the scoring function only checked an withdrawn flag, not the actual verdict text. Cross-examination looked adversarial but was cosmetic. Fixing this meant parsing the LLM's verdict ruling back into structured state before computing scores, not after.
  • Finding duplication across rounds: agents would re-state the same underlying claim each cross-examination round with a new finding ID, inflating finding counts (we caught one run where a single "this token isn't a password" argument generated 15 near-identical finding entries). Fixed with a two-pass dedup — by finding ID first, then by semantic similarity of line range + claim — so repeated arguments collapse into one tracked finding instead of accumulating noise.
  • WebSocket race condition: the backend often finishes generating the entire trial before the frontend finishes displaying it. Early versions let a completion message short-circuit the display queue, cutting off agents mid-sentence and jumping straight to the verdict. Fixed by deferring completion until the message queue is fully drained client-side.
  • Deployment: the original plan was Alibaba Cloud ECS, but ECS purchase required KYC verification with a passport or driver's license — documents I don't have (Indonesian KTP wasn't accepted). After confirming this was a hard blocker, not just a delay, we moved the backend to Render and kept Qwen Cloud as our Alibaba Cloud service usage.

What's next

  • Multi-file / multi-module review, so findings can span files (e.g., a hardcoded secret in config.py referenced unsafely in auth.py)
  • A persistent "track record" per codebase — repeat offenses (the same dismissed-then-reintroduced vulnerability) escalate in severity
  • Letting AXIOM and AEGIS query LEDGER directly mid-debate ("re-check line 94 — is that call user-controlled?") instead of arguing only from the original case file

Built With

  • fastapi
  • neon-postgres
  • next.js
  • python
  • qwen-cloud-api
  • render
  • vercel
Share this project:

Updates