Inspiration
Every AI code-review tool tells you it fixed your bug — and asks you to trust it. But a fix that looks right can still be bypassable, or can quietly break a feature. I wanted a system that doesn't ask for trust: it proves every claim by executing it.
What it does
The Courtroom: Appeals puts a Python file on trial. Over multiple rounds, AI agents prosecute it (write exploits), defend it (write patches), and appeal — and a real Nebius Token Factory sandbox decides what actually worked. An attack only counts if it extracts a secret the harness planted; a patch is only accepted if a sandbox run proves it blocks every known attack and keeps the app working. The guilt verdict is computed deterministically from sandbox facts — never voted by a model. A live Tavily search then grounds sentencing in real CVEs, and a three-model Nemotron "Supreme Court" rates severity.
How I built it
Nebius Token Factory hosts all inference via its OpenAI-compatible API, using NVIDIA Nemotron-3 models routed by role: Super and Ultra write competing patches in parallel each round; Ultra, Super, and Nano sit as a severity panel.
Token Factory Sandboxes run every exploit and patch in an isolated, network-disabled container and return real stdout/stderr/exit codes. This is the oracle — the harness plants a fresh random secret per run and only counts an attack if that secret returns through the app's own interface, so a model can't cheat by reading files directly.
Tavily provides live CVE/precedent grounding before the severity ruling.
The orchestration is a rounds loop with a growing exploit suite, deterministic stop rules, and patch-scoring (Bandit findings, then diff size).
Challenges I ran into
Non-determinism: The Nemotron-3 models are Mixture-of-Experts and aren't reproducible even at temperature 0, so eval outcomes vary run to run. I leaned into an objective sandbox oracle rather than model self-assessment, and added prosecution retries so a flaky reply can't be mistaken for "no vulnerability."
Reasoning models returning null content when the token budget was too small — I added guards and retries.
Keeping agents from grading their own work — I solved this by making the harness, not the model, decide success.
Accomplishments that I'm proud of
The sandbox is the judge, not the model. I built an oracle that plants a fresh secret each run and only counts an attack if it comes back through the app's own interface — so no agent can fake success or grade its own work. I verified the isolation directly: parallel runs can't see each other's files, and a run can't corrupt the base image.
A verdict you can trust. Guilt is computed deterministically from what actually happened in the sandbox, so it can never tie or be argued — LLMs are used only where judgement genuinely helps (rating severity), and their agreement or dissent is shown openly.
It genuinely hardens code across rounds. The attack → patch → appeal loop finds a bug, proves the fix, then attacks the patched code again — and the harness refuses any patch that breaks a feature or leaves an exploit open.
Real multi-model collaboration. Nemotron Ultra, Super, and Nano each play distinct roles — competing patch-writers and independent judges — rather than one model wearing every hat.
Grounded in reality. A live Tavily search surfaces real CVEs for the exact bug class (e.g. CVE-2024-12014, covering both path traversal and IDOR), anchoring the ruling to documented precedent.
Made reliable despite non-deterministic models. I turned the Nemotron MoE non-determinism from a blocker into part of the thesis — objective execution, not model confidence, is the source of truth.
What I learned
How to build a genuinely trustworthy agentic pipeline: separate the parts that need judgement (severity) from the parts that must be objective (did the exploit work, did the patch hold), and let real execution — not model confidence — be the source of truth.
What's next
Broader language and vulnerability coverage, accepting real GitHub pull requests as input, and exporting the exploit suite as a reusable pytest regression file.
Log in or sign up for Devpost to join the conversation.