What it does (project story)
Coding agents now produce code faster than anyone can review it. The failure mode of 2026 is not "the agent wrote nothing" — it is "the agent confidently delivered something that does not work," discovered later in production, in a demo, or in a judge's hands. Existing tooling optimizes generation; almost nothing optimizes trustworthy delivery.
VerifyGate is a pre-delivery verification harness for coding agents. A coding agent (or a human) submits a code change plus a task spec. VerifyGate copies the change into an isolated sandbox (scrubbed environment, symlink containment, bounded output (streamed through pipes into a capped in-memory tail buffer — never spooled to disk, so a check printing hundreds of megabytes cannot fill the temp filesystem while its own verdict is being decided), process-group timeouts, an interrupt sweep (SIGINT/SIGTERM stop the running check's whole process group too — a stopped run never leaves its check running unsupervised — and the CLI exits 130 with one line, writing nothing), and a bounded candidate — at most 10,000 verified files / 256 MiB, checked before any copy is made, so a vendored dependency tree cannot fill the temp filesystem one repair round at a time — and a bounded spec: at most 100 checks, each with its own capped timeout, since checks run sequentially and every result is sealed into the record, so verification mirrors that cap too), runs the spec's deterministic checks — unit tests, reproduction scripts, lint/type gates — and issues a verdict. Only artifacts that pass receive a signed, replayable evidence record: a hash-chained JSON document capturing exactly what was checked, the commands run, their outputs, and the resulting PASS/FAIL, so anyone can replay the same checks and confirm the same verdict.
The LLM — NVIDIA Nemotron via Nebius Token Factory — is used where LLMs are actually good: generating candidate patches in the repair loop (FAIL → model diff → apply to a copy → re-verify) and explaining failures in plain language. It is never the judge. PASS/FAIL is decided exclusively by deterministic validators: exit codes and assertions, not model opinion. Every real model call (model, purpose, token usage) is sealed into the evidence record's hash chain, so the token bill is as tamper-evident as the verdict. On failure, an optional Tavily search attaches current documentation references to the diagnosis; without a key, the degradation is sealed into the record rather than hidden. All five CLI subcommands share one exit-code contract (0 = PASS / intact, 1 = FAIL / tampered, 2 = usage or output error reported as a one-line message, never a traceback), Task specs are validated strictly before any check runs: a whitespace-only check command (which the shell would exit 0 on, sealing a vacuous PASS), a non-string command, or a boolean or absurd timeout is a usage error, never a run. And the clean-tree rehearsal ends with an installed-package smoke test: pip install the tree and run the verifygate console script from a different directory. A standalone verifygate verify command lets any third party recompute a record's chain and signature — and bind it to the spec file and candidate directory on disk — without re-running any check and without keys or network. Verification also enforces each record's structure (entry kinds are sealed into the chain from v2, and kind/order/required-field checks protect legacy records), and cross-checks a repair report's summary fields against its sealed attempts, so relabelling a FAIL check as commentary or flipping a stated final verdict fails the check. The cross-check also recomputes the report's quoted figures — the LLM usage total and the rounds used — from the sealed attempts, so rewriting the token bill fails verification too, and replay binds the supplied spec to the record's sealed spec hash (a different task running the same check commands does not reproduce a record). The sandbox copy is faithful down to permission bits: an executable script is verified as executable (never silently downgraded to a "Permission denied" FAIL), and the repair loop's patch applier honours git mode lines, so a fix that ships a script — created executable, or chmod'ed by a mode-only diff — is judged the way it was proposed. Those mode lines must name a regular file, though: any octal value was previously masked with & 0o777 at commit, so a "new file mode 120000" (symlink) creation silently produced a regular file holding the link target as text with mode 000 — and gitlink (160000) and directory (040000) modes did the same — while a malformed (non-octal) mode token was dropped silently and the patch applied without its mode line at all. Mode lines now validate at parse time: only regular-file modes are accepted, with their permission bits honoured exactly as before, and anything else is a clean PatchError rejection the loop feeds back. The applier also honours git rename and copy headers, which it previously ignored entirely: a pure rename or copy diff produced no file patch at all, and a rename with content changes was applied as a modification of the new path — which still lives at the old path — and rejected. Renames and copies are now parsed (from the git headers and from a header pair naming two different files) and applied in staging: hunks apply against the source content, the destination inherits the source's trailing-newline state, line-ending style, and permission bits, a rename stages the source as deleted, and the destination must be free under the same rules as a creation — an occupied destination or a context mismatch is a clean PatchError with the tree untouched. It also honours the deletion of an empty file, for which git emits no header pair and no hunks at all — only a "deleted file mode" line, which was not parsed: alone such a block failed as "no unified diff found", and inside a multi-file diff it was silently dropped while the rest applied and reported success, leaving a file the diff had deleted in the patched tree. The line is now parsed and flushed as a hunk-less deletion, and a hunk-less header pair naming /dev/null on either side stands as a creation or deletion on its own; a hunk-less deletion of a non-empty file is still a clean PatchError, and a failed deletion fails the whole diff atomically. A binary patch block is likewise never silently dropped: a "GIT binary patch" block (git diff --binary) and the "Binary files a/ and b/ differ" line carry no header pair and no hunks, so alone they failed misleadingly as "no unified diff found" and inside a multi-file diff they were silently dropped while the text changes applied and reported success. Both markers are now a clean PatchError at parse time, so the whole diff fails atomically and the loop feeds the rejection back; the markers match as whole top-level lines only, so a text patch whose content adds a line reading "GIT binary patch" still applies.
A numbered hunk is likewise held to its header: the parser read a hunk's body only up to the counts its header declared and never checked a shortfall, so a hunk declaring five old and five new lines but carrying one of each applied as if it had declared one, and a corrupt line inside a hunk silently ended it mid-body — dropping the rest of its lines, additions included — while the partial change applied. A numbered hunk whose body ends short of its declared counts is now a clean PatchError at parse time, so the whole diff fails atomically and the loop feeds the rejection back; lines beyond the counts stay tolerated by design (a committed real-mode patch carries a trailing context line beyond its counts, and replay reconstructs the counted hunk), and bare @@ hunks (which declare no counts) and stale start numbers are tolerated exactly as before. Text that can never cross its boundary is likewise rejected before any work: a check command carrying a NUL byte could never be spawned — commands reach the operating system as NUL-terminated strings — yet the spec accepted it, ran the checks before it, and then died mid-run with no evidence record for their decided verdicts; and a lone surrogate (a \ud800 escape decoded from JSON) is not valid Unicode and cannot be encoded as UTF-8, so it crashed evidence sealing after the checks ran — and crashed the chain verifier itself on a forged record, which surfaced as a usage error instead of CHAIN INVALID. The spec, sandbox, and evidence layers now enforce and mirror the same rules up front; adapter commentary carrying such characters is sanitised to U+FFFD instead of costing a decided verdict its record; a patch naming a NUL-bearing target path, or carrying surrogates in its text, is a clean PatchError the loop feeds back; and a forged record's rendered view still renders — the unencodable character degraded — because judging it is the view's whole job. The three values resolved from the environment — the HMAC signing key, the Tavily API key, and the configured Token Factory model id — are judged encodable as UTF-8 at the same validation that already judged their type and length: os.environ decodes a variable's bytes with surrogateescape, so a variable set by another process from non-UTF-8 bytes arrived holding lone surrogates while still being a string of legal length, and the signing key then crashed the HMAC encode at sealing and at verification alike; each is now the one ValueError at resolution, whichever route it arrived by. And the candidate tree hash is faithful to the bytes on disk: a filename is bytes, not text — a candidate may legally contain a name that is not valid UTF-8 (a Latin-1 name committed on another system; git tracks such names routinely), which the filesystem layer decodes with surrogateescape, and the workspace copy, the check execution, and the patch applier all round-trip those bytes — but tree_hash joined its lines as text and hashed them as UTF-8, which raises on the surrogate such a name decodes to, so a run over the tree lost the record its checks had decided and no record could be bound to the tree at all. Each hash line is now encoded as UTF-8 with surrogateescape, so a filename enters the hash as exactly its disk bytes: distinct raw bytes stay distinct (never collapsed into U+FFFD), and a surrogate-free tree hashes identically to before, so every committed record's candidate binding is unchanged. And a multi-file patch applies all-or-nothing: every file is validated against an in-memory staged copy first and disk is written only after the whole diff applies cleanly, so a diff whose second file fails never leaves the first file rewritten. That staging also judges the targets against each other: a diff staging both a file and a path beneath it (creating "x" and "x/y", in either order) used to stage cleanly and then fail in the commit loop — after the first target had already been written — with a raw OSError that escaped the repair loop's rejection handling and lost the whole report; staged targets may now never be ancestors of one another, so such a diff is a clean rejection with the tree untouched — with exactly the replacement exemption a real git diff needs: a file replaced by a directory (delete "x", create "x/y") and a directory replaced by a file (delete every file under "x", create "x") now apply, judged against the staged world plus the paths the same diff deletes and committed in phases, while a directory with any entry the diff leaves behind keeps the rejection. The applier is faithful down to the trailing newline too: the \ No newline at end of file marker is honoured on both sides of a diff, and a file's trailing-newline state changes only when a hunk actually reaches its end — a patch whose new side must end without a newline no longer silently gains one, and a patch that touches only the first lines of such a file no longer adds one at its end. And it is faithful to filenames: a header path was cut at its first space, so a patch for "my file.txt" targeted a file named "my" — a creation even silently wrote that wrong file — and git's C-style quoted paths for special or non-ASCII names were taken literally, so a real git diff touching such a file could never apply; paths now run to the line's end (or a TAB timestamp separator) and quoted paths are decoded before they are judged, traversal escapes included. Tamper handling is part of the contract, not an afterthought: every entry — the header included — is hashed over its full content (an extra field injected into a header previously verified INTACT), a record whose sealed token usage was rewritten into a malformed shape fails verification cleanly instead of crashing the verifier with a traceback, and a patch aimed at a binary file is a clean rejection the repair loop can feed back, not a crash. Verification also recomputes a record's consistency from its own parts — each check's verdict from its exit code and timeout, the record's verdict from its checks, a usage entry's total from its calls — so a record claiming PASS over FAIL checks, or quoting a token bill its calls do not add up to, fails verification even when its chain recomputes. A repair report's narrative is re-derived the same way: its stopped reason, round budget, patch linkage, preserved rejection, and credited adapter must be the ones its sealed attempts imply. Every attempt must also have been sealed under the same spec — the loop never changes the task, so an attempt recorded against a different task (same name, different checks) spliced into a report fails verification too. And a repair patch is held to the same size ceilings as a submitted candidate: the patch applier measures its staged result before writing anything, so a patch that would grow the candidate past 10,000 files / 256 MiB is an ordinary rejection the loop feeds back — never an oversize tree whose re-verification destroys the repair report it was part of.
The diagnosis references sealed beside a failure are bounded and normalised the same way — at most three, in the adapters' exact shape with per-field caps, normalised on the one sealing path — so a lookup returning malformed or flooding results can neither produce a record that fails its own verification nor bloat one. And the diagnosis text itself is bounded on the same path: at most 20,000 characters — roughly ten times the longest real diagnosis — truncated with the cut marked in the sealed text, so one runaway model reply (a 500,000-character explanation sealed a ~500 KB record intact before this) cannot bloat every copy of a record forever; verification mirrors the cap. A repair report's patch linkage is bounded the same way: the preserved patch text (stored truncated at 8,000 characters), the patch error strings, the changed-file lists, and each linkage's usage snapshot were judged only for type — inflating any of them with 500,000-character strings verified intact; verification now mirrors the source caps and generous ceilings over all of it, and a patch aimed at a target the filesystem cannot even stat (an over-long file name) is a clean rejection from the applier, never a raw OS error escaping the loop. And a check entry's own numbers are bounded the same way: its duration was checked only as a non-negative int and its exit code only as an int or None, so a record sealed over a duration of 10**12 ms, a timeout entry carrying an exit code, a completed check carrying none, or an exit code of 999,999 all verified intact; verification now mirrors the sandbox's emission contract — duration at most twice the longest legal timeout in milliseconds, exit code None exactly when the check timed out, otherwise a real int in -255..255. And the run mode itself is validated at every entry point, not only by the CLI's argument choices: before this, a library caller passing any other mode — a typo, an empty string, or a shell fragment — ran every check and sealed a record that failed its own verification (the header must name a real mode, and the replay command interpolates the mode raw — the fragment form smuggled a second command into the very string a reader is told to copy and run), a repair delivered a whole report of such records, and the CLI's adapter factory silently treated any non-judge mode — including one read from a record's own untrusted header on the replay path — as real mode, selecting paid adapters instead of reporting a usage error. The runner, the repair loop, replay, and the adapter factory now all reject an invalid mode before any work happens. And the labels that name the spec and candidate in a record's replay command are validated at the same entry points: they were the last caller-supplied run parameters with no validation — a non-string label crashed the quoting helper with an unhandled TypeError after the checks had already run (and mid-replay, past its error handling), an empty label sealed as a quoted empty path, and a 500,000-character label sealed a 500,000-character replay field that verified intact, because the metadata rules capped no replay length. Both labels are now non-empty strings of at most 1,000 characters, validated before any work, and verification caps the replay string itself at 16,384 characters — the largest replay string validated labels can produce. The real adapters' own constructors are held to the same standard: a timeout_s that is a string, a bool, zero or negative, infinite, or effectively unbounded, a transport that is not callable (previously selected with or, so a falsy non-None value was silently replaced by the real, paid transport), and a Tavily api_key of a non-string type interpolated into the Authorization header all construct silently no longer — each is a ValueError at construction, beside the configured-model validation, before any call is made. The evidence signing key is held to the same standard: it was resolved with a bare environment fallback that applies only when the variable is unset, so a set-but-empty VERIFYGATE_SIGNING_KEY — the classic CI/deployment mistake — silently became an empty HMAC key while the record disclosed that a custom, non-dev key had signed it. The resolved key is now validated wherever it is used — at sealing, at verification, and in the runner before any check runs: 8..512 characters, not whitespace-only, no control characters, and the error names only the rule and the length, never the key itself. And the documents the judge-facing tools read are held to the rule a tampered record must fail, never crash, its judge: verify, replay, and the web view assumed a record's entries were a list of objects — an assumption the chain check itself never made — so a record rewritten with a list of strings, a bare string, an object, or a number for its entries crashed verify and replay with a traceback out of the shared header helper, and crashed the web view on half a dozen more malformed shapes. Every consumer now guards the shapes it reads: a malformed record fails cleanly — CHAIN INVALID from verify, not reproduced from replay, and a rendered page carrying its CHAIN INVALID banner from the web view. Output destinations are protected the same way: a --out naming the task spec (or, for the web view, the record itself) is a usage error, and repair --patched-out may not equal, contain, or sit inside the original candidate tree — a repair never modifies its original, not even by being told to copy the patched tree over it. The protection is judged by file identity, not just by path: a --out given as a hardlink alias of the spec or the record overwrote it through the alias, and a patched-out destination holding a hardlink to (or a symlink at) a candidate file was written through by the copy — file outputs now also match by inode, and the destination tree is scanned before any work for entries aliasing the spec, the report path, or the candidate tree. The patch applier's commit is judged the same way: a patch target hardlinked to a file outside the workspace was rewritten through the shared inode, modifying a file the patch never named — the commit now unlinks a multiply-linked target before writing, so the patched path gets its own inode and every other name keeps its content and mode. The patched-out copy is judged once more, at the moment the patched tree exists: a destination entry on a path the copy will write that is a symlink, or a regular file hardlinked to another name, carried the write outside the destination tree to an unrelated file the alias scan does not protect — those written paths are now scanned immediately before the copy, a write-through entry is an output error with nothing copied, and aliases on unwritten paths are still not judged.
A file output is judged against the candidate's contents, not just its directory: a --out naming an existing candidate file — directly, through a hardlink alias, or through a symlink — overwrote that file with the evidence record and exited 0, sealing a tree hash for a candidate the invocation had just destroyed. File outputs are now judged by inode against every entry of the candidate tree, before any check runs; a new file inside a candidate stays allowed, and the dogfood record is regenerated by staging its run output outside the candidate and moving it into place. The environment scrubber itself is validated the same way: sanitized_env — the one function that builds the environment every check runs with — iterated its base on trust, so a non-mapping base or a non-string key crashed it with a raw AttributeError, and an allowlisted entry with a non-string value ({"PATH": 123}) passed through verbatim, making subprocess.Popen raise its own raw TypeError inside the execution primitive: the scrubber produced an environment that broke the run it was meant to make safe. A non-mapping base is now one ValueError, and entries whose key or value is not a string are dropped exactly the way non-allowlisted names already are, so the scrubbed environment is always a dict[str, str]. The sealing function itself is validated the same way: build_record — the one function that assembles and seals a record — computed over its arguments on trust, so a malformed llm_usage crashed inside its own usage sum (a non-object call or usage with an AttributeError, a string token count with a TypeError), a negative token count sealed a usage entry the consistency mirror then rejects, and a non-object check entry or diagnosis crashed on the mapping spread. The shapes it touches are now validated before anything is sealed — one ValueError — while field-value consistency (a verdict against its checks, the caps, the header's mode and hashes) deliberately stays the verification mirror's job. The adapters' optional surface is guarded the same way: a duck-typed adapter's name and its usage-log methods (drain_calls / pending_calls) were read directly by the runner and the repair loop, although the validators promise both stay optional — a nameless adapter crashed a FAIL run while the diagnosis entry was being assembled, and a usage log that raised destroyed a decided verdict's record and a completed loop's report. Names are now read through getattr (a missing name becomes the fallback label) and the log only through guarded helpers: a missing, broken, or non-list usage log degrades to no recorded usage, never to a lost record or report. The candidate traversal drops unresolvable symlinks instead of crashing on them: the single traversal behind the workspace copy, the tree hash, and the repair snapshot resolved every link with a bare Path.resolve(), so a symlink loop — a link pointing at itself, or two links pointing at each other — raised RuntimeError out of the traversal and killed the run (at the CLI, an adapter error with no record), even though the candidate's real files were fine. A looping link is now dropped exactly like a dangling, outside, or directory link, and the candidate still gets its verdict. Check commands run with a closed stdin: they inherited the harness's own stdin, so the same candidate and the same check (read answer; test "$answer" = yes) sealed PASS when the harness's stdin happened to carry yes and FAIL when it did not — a verdict depending on input that is in neither the spec nor the candidate, and that replay (a different process, different stdin) could never reproduce — and a check that consumes stdin hung until its timeout whenever the harness's stdin was a terminal or a held-open pipe. Checks now run with stdin attached to /dev/null: reads see immediate EOF, the verdict is a function of the spec and the candidate only, and a check can never consume input meant for the harness or the operator. And a deeply nested candidate still gets its verdict: the candidate traversal and the workspace directory skeleton walked the tree by recursion, so a candidate nested deeper than Python's recursion limit crashed the copy, the tree hash, and the repair snapshot with a bare RecursionError, and shutil.rmtree raised the same error out of workspace cleanup, leaking the copy. The traversal, the skeleton, and workspace removal (a new iterative sandbox.remove_tree, used by the runner, the repair loop, replay, and the CLI) now carry their stacks on the heap, in the same sorted depth-first order as before. And no process outlives its check: the process-group SIGKILL that reaps a check's children ran only on the timeout path, so a check that exited normally while leaving a background child behind leaked that child onto the host — it outlived the run, could write into the workspace being deleted, and could still be alive when replay re-ran the same checks. The group kill now runs after every check, pass or fail (a child that detaches into its own session escapes it, exactly as it already did on the timeout path — the documented limit of this sandbox). And the JSON loader never blocks on a special file: every spec and record is loaded through one bounded loader, which opened whatever path it was given — so a record or spec path naming a FIFO hung verify / view / replay indefinitely, with no timeout, on the artifact those commands exist to judge. The loader now stats the path first (following symlinks, so a symlink to a regular file still loads) and rejects any existing path that is not a regular file as a clean usage error; a missing file still raises OSError. And no write blocks or vanishes on a special file either: every file VerifyGate writes — the evidence record, the repair report, the rendered HTML view — went through a bare Path.write_text, so a FIFO destination blocked the write indefinitely after the verdict was already decided, and a device destination such as /dev/null silently discarded the document while the CLI reported it written. All three writes now go through one write path that stats the destination first (following symlinks) and rejects any existing destination that is not a regular file as a clean output error; repair --patched-out likewise judges an existing non-directory copy destination before any work runs, instead of running the repair to completion and only then failing at the copy. And a JSON document cannot crash its judges by nesting: the loader's decoder recurses, so a record or spec nested ~10,000 levels deep — tens of kilobytes, far under the byte ceilings — died with an unhandled RecursionError traceback on every consumer at once. The loader now converts that into the same clean usage error as malformed JSON, and rejects any document nested deeper than 100 levels (genuine documents nest about ten), measured iteratively after decoding.
And the decoder rejects duplicate object keys: it silently kept the last value for a repeated key, so a record whose text states "verdict": "FAIL", "verdict": "PASS" verified INTACT — this loader read PASS (last wins) while a first-wins parser, or a human reading the raw text, reads FAIL: the same artifact judged two different ways, on the very field the record exists to seal. A repeated key, at any nesting level and including keys that only collide after \uXXXX decoding, is now the same one-line usage error as malformed JSON; every document VerifyGate writes is built from a dict and can never contain a duplicate. The Token Factory adapter's configured model is resolved the same strict way: only an explicit None means "not supplied" and falls back to NEBIUS_MODEL / the default, so a supplied-but-falsy model id ("", 0, False) is a usage error at construction instead of a silent swap to a model the caller never named. The CLI's own optional arguments are held to the same rule: --out "" on run / repair / view wrote the default output path instead of the one the caller named, repair --patched-out "" was silently ignored, and verify --spec "" verified with no spec binding at all and reported RECORD INTACT — a caller who believed they had bound the record to a spec had bound it to nothing. An explicitly empty value is now a usage error (exit 2) for each of these; only an omitted argument means "use the default / no binding". The execution layer is held to it too: sandbox.run_command — the one primitive every verdict passes through — validated its arguments only transitively, through a validated spec, so a direct caller got a TypeError for a non-string command, a vacuous exit-0 result for an empty/whitespace command (nothing checked, everything "passed"), and string/bool/None/ zero/float timeouts crashing, shrinking, waiting unboundedly, timing out instantly, or breaking the spec's int-only rule; a cwd of 123 was stringified into a lookup for a directory literally named "123". The primitive now enforces exactly the spec's rules as a ValueError before any process spawns, and validators.run_checks validates its spec and workspace at its own entry. The adapter methods are held to it too: the adapters themselves were validated at the entry points, but generate_patch / explain_failure took their arguments on trust — a non-string brief crashed the mock with an AttributeError and was silently stringified by the real adapter into a paid prompt ("Task brief: 123"), a non-dict or non-serializable failing_evidence crashed before the call, and a non-list check_results or an entry missing verdict crashed the explanation. A non-string lookup query crashed the mock lookup — whose contract is must not raise — and was sent verbatim to the Tavily API. The LLM methods now raise one ValueError for a malformed argument before any model call (an entry missing verdict simply is not a failure), and lookups degrade on an unusable query exactly like a failed lookup. And the per-call bound overrides themselves are validated: every ceiling can be tightened per call (candidate files/bytes in the bounds check and the patch applier, the JSON loader's max_bytes, the reference normaliser's limit), and those overrides were the last numeric limits taken on trust — True silently ran as a bound of 1 (in the JSON loader it even named itself in the error, "exceeds the maximum of True bytes"), a float compared as-is, a string crashed with a TypeError (inside the patch applier not a PatchError, so it escaped the repair loop's rejection handling; in the JSON loader it crashed in max_bytes + 1), a negative bound produced a nonsense rejection, and an enormous override silently loosened the very ceiling it overrode. An override may only tighten a bound now — a real int within the ceiling in force, else one ValueError before any file is counted, staged, written, opened, or normalised. And the one sanctioned post-seal edit is validated the same way: update_replay_command — the repair loop's rewrite of a patched attempt's replay field, re-sealed under chain v3 — took both arguments on trust. A command that was not a string, was empty, carried a shell fragment, named the wrong mode, or ran to 500,000 characters was written and re-sealed into a record that fails its own verification; a malformed record crashed with TypeError / KeyError / IndexError after the field had already been overwritten. Both arguments are now validated before anything is mutated — the command against the two emitted templates and the header's mode, the record for the shape the re-seal reads — as one ValueError, with the record left byte-for-byte unchanged. And the verdict aggregation itself is validated the same way: overall_verdict — the one function that turns per-check results into the sealed verdict — took its input on trust. An empty iterator aggregated to PASS (results and all(...): an iterator is always truthy, and all of nothing is true) while an empty list aggregated to FAIL — a vacuous PASS from the aggregator itself; None silently aggregated to FAIL, a verdict for checks that never ran; a string or a list of plain dicts crashed with an AttributeError. And each result's verdict field was trusted blindly: a result claiming PASS with exit code 3, or PASS while timed out, aggregated to PASS, against the one derivation rule (a check passes iff it exits 0 before its timeout) — and the record sealed from it failed the consistency mirror. The input is now materialised and every entry validated before aggregation: a CheckResult whose verdict-bearing fields are the shapes run_checks emits, and whose verdict is exactly the verdict those fields derive. An inconsistent result is one ValueError, never a verdict. And the patch applier's root is validated the same way: apply_patch coerced the directory a patch applies under with a bare Path(...) — a non-path root crashed with a raw TypeError (not a PatchError, so it escaped the repair loop's rejection handling), an empty string silently meant the current directory (whose whole tree the bounds check then measured as the candidate), and a missing root was silently created by a creation diff, materialising a tree wherever the path pointed. The root now goes through the same candidate-directory validation as every other entry point, before anything is parsed, staged, or written: one ValueError for a malformed root, the usual FileNotFoundError for a path that names no existing directory, and the root is never created.
And the candidate traversal itself is validated the same way: iter_candidate_files — the single traversal behind the workspace copy, the tree hash, and the repair loop's prompt snapshot — coerced its root with a bare Path(...), so a direct caller got a raw TypeError for a non-path root, an empty string silently traversing the current directory, and a NotADirectoryError for a regular file instead of the one FileNotFoundError. The root now goes through the same candidate-directory validation, eagerly at call time even though the traversal returns an iterator: one ValueError for a malformed root, the usual FileNotFoundError for a path that names no existing directory, and the containment semantics unchanged. And the record writer is validated the same way: write_record — the one function that materialises a sealed record to disk, and the CLI run handler's only write path — converted its destination with a bare Path(...) and took its record on trust, though the loaders had validated their paths since v1.46.0. A non-path destination crashed with a raw TypeError, an empty string surfaced as an IsADirectoryError for the current directory, a non-object record was silently written as a JSON file no VerifyGate consumer accepts (overwriting whatever already sat at the destination), and a non-serializable record escaped as a raw TypeError past the CLI's OSError-only write handling. The destination now goes through the loaders' own path validation, the record must be a dict, and it is serialised before the file is touched, so a rejected record leaves a pre-existing destination byte-for-byte unchanged — one ValueError, which the CLI reports as its one-line output error. And the diagnosis error notes are normalised the same way: the runner's _cap_error helper truncated a long string but copied any other value verbatim — and the one note it does not format itself, a lookup's last_error, is adapter-supplied, so a third-party lookup may set it to anything at all. An int or dict note sealed verbatim produced a record whose diagnosis entry fails the structure check, so the record failed its own verification the moment it was written; a non-serializable note (an exception object, a set) crashed the record build's JSON encoding with a raw TypeError and the FAIL record was lost entirely. A non-string note is now discarded for a deterministic placeholder naming only the offending type, and the explanation helper raises one ValueError for a non-string input instead of crashing in len(); string notes and explanations pass through exactly as before. And replay judges a report's patch linkage by shape before it reads it: the reconstruction path assumed each later attempt's patch_from_previous is an object and read it with .get, so a linkage rewritten as a bare string, a number, a list, or true crashed replay_document with an unhandled AttributeError — past the per-record path whose doctrine is that a malformed record replays as not reproduced, never as a crash (and at the CLI the backstop reported it as a usage error instead of the not-reproduced answer). A non-object linkage is now simply no applied patch recorded: the attempt cannot be reconstructed, and replay reports exactly that. And verification judges a report's credited adapter without trusting its shape: the narrative check's sealed-adapter cross-check built a set out of the stated llm_adapter, so a stated adapter rewritten as a list or an object — unhashable values — crashed verify and the web view with a TypeError (at the CLI, a usage error instead of the tampered-report answer). The comparison now requires a string adapter equal to the sealed names; any other stated value is a mismatch by construction.
Judge Mode runs the entire flow with zero keys and zero network: a deterministic mock adapter supplies the words, while every validator still runs for real. A second bundled sample in JavaScript (Node's built-in test runner as the checks, no Python inside the candidate) proves the harness is language-agnostic: PASS, FAIL, and a Judge Mode repair FAIL -> PASS, with committed evidence records — and the same holds in real mode, where a committed report shows Nemotron 3 Super 120B repairing the Node candidate FAIL -> PASS with its own fix (2 calls, 6,390 tokens sealed). VerifyGate also dogfoods itself — a committed spec runs its own 2921-test suite as a VerifyGate check, and the PASS record ships in the repo. The hosted demo renders both real-mode repair reports (Python and JavaScript) with their chains verified on the page — including the chain-v3 reports, rendered and linked from the landing page since v1.103.0 (before that the demo rendered only the legacy chain-v1 reports). And a record can be replayed, not just verified: verifygate replay re-runs the checks and compares verdicts, reconstructing each patched attempt from the original candidate plus the patch recorded in the report — all five committed real-mode repair reports reproduce exactly this way in Judge Mode, zero keys. The newest of them are sealed under chain v3: Nemotron 3 Super 120B repaired the Python sample FAIL -> PASS after its first diff was rejected and fed back (3 calls, 9,407 tokens sealed), and repaired the JavaScript sample FAIL -> PASS in one round (2 calls, 5,318 tokens sealed) — real mode under the newest chain is proven in both languages. Chain v3 also covers the record's own top-level metadata — the replay command a reader is told to run, the default-dev-key disclosure, the version, and the timestamp — so none of them can be rewritten after sealing; legacy records are held to matching metadata consistency rules at verification time. The real adapter validates every decoded Token Factory response field by field before recording anything: a malformed shape (a message that is not an object, a non-string content, token counts that are not non-negative ints) is a clean adapter error, never an unhandled traceback and never a nonsense usage entry sealed into a record. The one recorded field that validation missed — finish_reason, copied verbatim — is now normalised at decode time like the model name: a non-string or over-long finish reason (an int, an object, a 500,000-character string) is recorded as None rather than sealed into a record that fails its own verification.
And the sealing path itself no longer trusts adapter-supplied metadata: the runner — and the repair loop, for its report-level copies — normalise adapter/lookup names and recorded usage calls through shared helpers, so an adapter with an empty or 500,000-character name, or one recording a malformed call (a 500,000-character model string, an int finish reason, a usage that is not an object at all — which used to crash the record build and lose the record) still seals a record that verifies; a configured Token Factory model id is likewise rejected at construction when it is empty, not a string, or over the sealed model cap, instead of reaching the record through the fallback path. And the files themselves are bounded before any field is judged: every loader used to read its JSON input in full with no size bound, so a spec file padded to 5 MiB ran to PASS and an evidence record padded to 70 MiB verified intact — a hostile file could exhaust the verifier asked to judge it before a single field check ran. Spec files (4 MiB) and record files (64 MiB) now load through a bounded loader that reads at most the cap plus one byte and rejects an oversize file as a clean usage error; both caps sit far above what a document at every field ceiling serialises to. And a patch the loop applies is a patch the report carries, in full: the loop used to apply a proposal of any size while storing only its first 8,000 characters, so a valid 9,413-character diff applied, passed, and verified intact while replay could not reconstruct its attempt from the truncated fragment. The patch text is now bounded at the applier, before parsing, by one generous ceiling (65,536 characters) that is also the stored cap: an oversize proposal is a clean rejection — preserved, fed back, retried within the round budget — and every applied patch is stored whole and replays exactly. And the diagnosis can never hold a verdict hostage: the explanation is commentary, so when the explanation call itself fails the run seals a deterministic fallback rendered from the real check results plus a diagnosis_error note — before that guard, one failed explanation call cost a FAILed candidate its entire evidence record, aborted the repair loop on its first attempt, and made a real-mode replay report "not reproduced" when only the words had failed. The same holds for patch generation: a generation failure (an outage, a malformed response, an empty completion) ends the repair loop with the report delivered — attempt 0's sealed FAIL record, the failed round counted, the failure preserved with error_kind: generation_failed and stopped_reason: generation_failed — instead of the exception destroying the whole report; verification re-derives that stop from the preserved failure kind, so a generation failure cannot be rewritten as an applier rejection, or vice versa. And the record VerifyGate seals always verifies — including when the caller's own paths are unfriendly: spec/candidate labels are interpolated into the sealed replay command through a quoting helper (bare path tokens unchanged, anything with a space, a quote, or a non-ASCII character single-quoted in shlex.quote form), after a round where a checkout under a spaced or Chinese-named directory sealed a record that reported CHAIN INVALID the moment it was written; the emitted replay command is pinned by tests that execute it — and by one showing a $(touch …) path cannot weaponise it. The repair loop's round budget is validated the same strict way as a spec timeout: before this, a boolean budget produced a report whose stated max_rounds fails verification's own non-bool-int rule, a float budget (2.5) silently allowed three rounds, a string crashed with a TypeError, and --max-rounds 1000000000 was accepted verbatim even though every round is a paid model call — the budget is now a real int in 0..10 at the library and the CLI, and verification rejects a report claiming a budget above the loop maximum.
How we built it
Python 3.12, standard library only for the harness (subprocess orchestration, hashlib hash-chained evidence records, HMAC-style signing, a hardened unified diff applier). Nebius Token Factory (OpenAI-compatible) drives NVIDIA Nemotron in two places only: patch generation (Nemotron 3 Super 120B for the committed FAIL→repair→PASS report: 2 calls, 5,367 tokens sealed; the JavaScript sample's real-mode repair report: 2 calls, 6,390 tokens sealed; a chain-v2 Python repair report: 3 calls, 11,726 tokens sealed; a chain-v3 Python repair report: 3 calls, 9,407 tokens sealed; a chain-v3 JavaScript repair report: 2 calls, 5,318 tokens sealed) and failure explanation (Nemotron 3 Nano 30B for the committed real-mode FAIL records: 1 call, 1,133 tokens sealed; a chain-v3 successor FAIL record: 1 call, 1,108 tokens sealed, its diagnosis entry carrying the new diagnosis_error: null beside the sealed Tavily degradation). Tavily is adapter-isolated on the diagnosis path with graceful, in-record degradation. The evidence web view is a single self-contained HTML page (no JavaScript) that recomputes the hash chain at render time and shows a CHAIN INVALID banner for tampered records.
Challenges / what we learned
Nemotron's reasoning models can return empty completions at a token ceiling and produce diffs that almost apply — merged context lines, bare @@ hunks. They can also deliberate inside the reply content itself: one Super 120B run spent its entire 8,192-token budget reasoning in the open (finish_reason=length) and never emitted a usable diff, so the patch prompt now orders silent reasoning and names that failure explicitly. VerifyGate's answer is structural: a diff whose context does not match the file on disk is rejected, never forced, and the applier's exact rejection is fed back to the model for the next round. Generation stays probabilistic; verification does not. The same discipline marks a current limit: on a live task whose fix was deleting a file outright (a statistics.py stub shadowing the standard library), both Nano and Super failed to reproduce the whole file byte-for-byte in a deletion diff — escaped docstring quotes first, then collapsed blank lines — across five attempts and 64k tokens, and VerifyGate rejected every attempt rather than force a deletion it could not verify. The patch prompt now teaches the create/delete file forms explicitly, which fixed the escaping failure; verbatim whole-file reproduction remains the models' weak point, and the harness treats that as the model's failure, not a reason to weaken the check.
The same pattern repeated on the first live task whose fix was a permission change alone (a correct script missing its executable bit): Nano spent three rounds and 22,697 tokens proposing content edits — including a chmod line inside the script, which can never run because the missing permission is what prevents the script from running — so the prompt now also teaches the mode-only diff form, and the next live run repaired the task FAIL -> PASS in one round with the exact mode-only diff (2 calls, 2,106 tokens sealed in the run-log).
Built with
python · nebius-token-factory · nvidia-nemotron · tavily · pytest · html
Token Factory feedback
Token Factory's OpenAI-compatible endpoint and the Nemotron 3 family made integration straightforward; model IDs are configuration, not constants. Two developer-experience notes: (1) reasoning models can consume the whole token budget in hidden reasoning and return an empty completion with finish_reason=length — surfacing a reasoning-token count in the usage object would make this diagnosable from the response alone; (2) the Nano model's diagnoses were excellent while its unified-diff formatting was unreliable at smaller budgets — the Super 120B model was markedly more reliable for diff generation. Overall the credit and model range were more than enough to build and iterate: total sealed usage across the committed evidence is 40,449 tokens, plus roughly 39k exploratory tokens during development and ~15.6k in one aborted chain-v2 repair attempt whose diffs were all rejected — the run that exposed the in-content deliberation failure described in Challenges above.
Log in or sign up for Devpost to join the conversation.