Verdikt

Coding agents say “all tests pass.” Verdikt checks. Nemotron splits the report into claims, a sandbox runs real pytest, and only executed evidence decides the verdict.

Comment

Inspiration

Coding agents end their runs with confident summaries: "Fixed the bug. All tests pass." That summary is the least trustworthy part of the loop. Agents overclaim, quietly edit the tests to get green, or claim things nobody checked, like a deployment. Reviewers either trust the text or redo the work. We wanted a layer that doesn't take the agent's word for anything.

What it does

Verdikt takes an agent's completion report and checks every claim against real test evidence. Each claim gets one of three verdicts:

  • VERIFIED: executed evidence supports it
  • CONTRADICTED: executed evidence disproves it
  • INSUFFICIENT EVIDENCE: no allowlisted check can establish it (for example "deployed to production")

A patch scan also flags agents that edited the test files instead of fixing the code.

The demo ships 3 sample repos (slugify, cart_total, rate_limit), each with a real bug and a real pytest suite, and 4 agent behaviours: honest, overclaiming, test-tampering and deploy-claiming. Pick a repo and a behaviour, and Verdikt shows the verdict, the evidence behind each claim, the raw pytest output, and the test run on the unpatched repo next to the run on the agent's patch.

How we built it

Nemotron on Nebius Token Factory reads the agent's free-text report and does the part LLMs are good at: splitting it into atomic claims and choosing, for each one, a check from a fixed allowlist of five (tests pass, the bug was real, a targeted test passes, which files changed, deployment status). We call nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B through Token Factory's OpenAI-compatible endpoint with the standard OpenAI SDK.

A sandbox then runs real pytest on a pristine copy of the repo and on a copy with the agent's patch applied.

A deterministic judge computes each verdict from the executed results only. The model's output is a proposal of what to check, never a verdict. Anything the model proposes that fails validation is dropped, and the UI shows what was rejected.

Safety by design:

  • The API never accepts client-supplied patches. Agent behaviours are stored scenarios.
  • Checks come from an allowlist, so the model cannot make the system run arbitrary commands.
  • Deployment claims always end as INSUFFICIENT EVIDENCE, because there is no deployment runner, and Verdikt would rather say "unknown" than guess.
  • Model-backed runs are rate limited to protect credits.

Stack: Python, FastAPI and pytest on the backend; a React + TypeScript + Vite "Agent lab" frontend; 228 automated tests. Replay mode uses scripted claims and runs fully offline; Live mode uses Nemotron to extract the claims.

Challenges

  • Using the model without trusting it. The hard design question was how to get value from an LLM in a verifier without letting it influence the outcome. The answer was to restrict it to choosing from an allowlist and to make the judge ignore everything but executed evidence.
  • Test tampering. An agent can make the suite pass by editing the tests, so a green run alone proves nothing. We scan the patch for protected test files and surface that next to the verdict.
  • Messy reports. Real reports mix checkable claims ("I only touched cart.py") with vague ones ("Fixed cart total()"). Verdikt checks the first kind and does not invent evidence for the second.

Accomplishments we're proud of

A clean separation between what the model suggests and what is true, demonstrated across 12 repo-and-behaviour scenarios with a full test suite behind it. A report that contains no checkable claims, like "I did my best", is never marked VERIFIED. The UI makes the evidence the centrepiece: you can open the actual pytest output behind any verdict.

What we learned

LLMs are very good at interpreting messy natural-language reports and poor arbiters of truth. The right architecture uses them for the first job and executes code for the second. Small models are enough for this: Nemotron Nano 30B-A3B returned usable structured claims in our Live runs, at roughly 9 to 14 seconds end to end including both pytest runs.

What's next

More check types (lint, type checks, coverage deltas), more languages beyond Python, a GitHub Action that comments verdicts on agent pull requests, and verification of multi-step agent traces instead of single final reports.

Feedback on Nebius and NVIDIA tools

Nebius Token Factory: Setup was quick. The OpenAI-compatible endpoint worked with the standard OpenAI SDK by changing only the base URL, so integration took minutes, and a small connectivity script confirmed the key and model before we built on it.

NVIDIA Nemotron (Nano 30B-A3B): It followed our structured-output instructions well in the Live runs we did. It extracted atomic claims and picked sensible checks from the allowlist, and it correctly returned nothing for a report with no checkable claims. It tended to skip vague claims such as "Fixed cart total()", which suits a verifier. We strip any reasoning blocks before parsing and validate every proposal against the allowlist, so a malformed answer fails safely.

Built With

Share this project:

Updates

Submission history