Attest

A coding agent that must prove its patch: sealed tests, adversarial checks and an offline-verifiable Proof Bundle before you merge.

Comment

Inspiration

Coding agents already write plausible patches in seconds. What a reviewer still lacks is evidence: 66% of developers name "AI solutions that are almost right, but not quite" as a frustration with AI tools, the most common answer in the Stack Overflow Developer Survey 2025. We built the agent around the reviewer: it does not get to say "done"; it has to show why.

What it does

You give Attest a Python repository at a pinned commit and a precise request. Then:

  1. Specify. NVIDIA Nemotron 3 Ultra turns the request into a specification and an interface contract; when the request names an outside standard, the specification is grounded with Tavily search and the bundle cites the sources. A contract auditor looks for requirements that cannot all hold, and only a contradiction demonstrated by a witness program that Attest executes ends a run as INFEASIBLE.
  2. Seal. A separate Nemotron 3 Super instance writes acceptance tests before any code exists, with a coverage map that is checked for completeness. The tests are sealed with a digest, and they must fail on the untouched repository, by assertion, before they count. The implementer never receives them.
  3. Build. The implementer (Nemotron 3 Super) acts only through JSON actions (search, read, replace, write, run tests, finish), each checked against its schema and the write scope, on its own branch of a Nebius Token Factory Sandbox checkpoint.
  4. Break. A supervisor that never imports the candidate's code verifies it in a fresh sandbox with networking off: every sealed test in its own process, bound to a nonce and the sealed file's digest. Generated one-line mutants run in parallel sandboxes; the recorded mutation gate measures how many the sealed tests catch, and flags cases with no mutable sites. A second model writes adversarial tests; a drift audit compares the action trail with the request; an arbiter classifies every failing generated test, and a corrected test counts only after mechanical validation.
  5. Verdict. One recorded rule, in code, turns the gates into VERIFIED, NEEDS HUMAN (with the reason), INFEASIBLE, NO CHANGE or ERROR. The evidence ships as a Proof Bundle (exhibits, the counterexample, the per-call ledger, a manifest with every file's sha256), and attest check re-derives the verdict offline in under a second.

Reliability is part of the evidence: when an operation's outcome is uncertain (a sandbox answering an error after the request was sent), the run and its batch stop, and uncertain operations are not automatically replayed. A batch resumes only after read-only checks and recorded reconciliation notes naming the affected item ids; console runs remain held until an operator records a reconciliation note and clears the hold. This reduces duplicate-execution risk; it is not a guarantee about provider billing. Per-run and daily inference budgets are enforced before dispatch; Sandbox, search and hosting charges are separate.

The film starts a new Nemotron run on console 0.10.3. It ends NO CHANGE after the implementer exhausts its 12-turn budget without a patch. That outcome is shown as recorded; the subsequent VERIFIED run and refusal are separately labelled records from the pinned 0.10.0 evaluation below.

How we use NVIDIA Nemotron and Nebius

  • Nemotron 3 Ultra (nvidia/Nemotron-3-Ultra-550b-a55b): specification, contract audit, drift audit, arbitration.
  • Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b): sealing, implementation, breaking (adversarial tests).
  • Nebius Token Factory: every call is a structured-output chat completion (strict JSON schema), reasoning bounded with reasoning_effort per role, metered per token in the run's ledger (the console shows each call's model, tokens and cost).
  • Nebius Token Factory Sandboxes: one retained checkpoint per task; disposable branches, networking off, for the red check, every verification, every mutant and every adversarial run; operation ids recorded in the bundle so every step can be traced.
  • Tavily: grounding for requests that name an outside standard (recorded run of 15 Oct: the semver task, 2 searches, 6 cited sources including semver.org).

The historical Tavily-grounded run ended NEEDS HUMAN; it demonstrates integration, not a verified patch. The final 21 Oct evaluation made no Tavily calls. Recorded grounding evidence.

Results: the 21 Oct 2026 evaluation of version 0.10.0 on Nemotron

The frozen suite and method, 8 tasks x 2 systems x 2 attempts, every scheduled cell reported, judged by held-out tests that neither system saw. The plain agent is the same implementer with the same model, tools and turn budget and none of the gates.

Plain agent (16 cells) Attest (16 cells)
Correct patch accepted 8 2
Broken patch shipped 2 0
Correct patch held for a human 1 2
Correct refusal of the task that cannot be satisfied 0 2
Hold without a patch, or unfinished 1 4
Sandboxes error after a request was sent: stopped, never replayed 4 6
Committed inference $0.34 $1.04

On the task that cannot be satisfied the plain agent shipped a broken patch both times; Attest refused both times, once with an executed witness of the contradiction (an earlier version had shipped both on 17 Oct). The price is caution: Attest accepted 2 correct patches to the plain agent's 8, and three of its holds came from its own test writer failing to produce a valid sealed suite. An Attest run costs about 6 cents of Nemotron inference (5.8 cents on average over the 10 runs not stopped by a sandbox error). Report: full evaluation report; the earlier evaluation: earlier evaluation report.

Challenges we ran into

The verifier is where the engineering is: a supervisor that never imports candidate code, results bound to nonces and digests, mutants that only count when an authentic test failure kills them, a correction that only counts after it is validated by execution. On the platform side, a request whose outcome is unknown is the hard case: during the evaluation the Sandboxes API answered six spawns with HTTP 500 after they were sent; Attest stopped and required reconciliation. Read-only operation listings with positive controls found no matching spawn, including a follow-up at least one hour after the last failure. The remaining cells resumed after the affected items were acknowledged by id; failed cells stayed in the results and were not re-run.

What we learned

Honest numbers beat flattering ones. Our first evaluation showed Attest shipping the same two broken patches as the plain agent; the fix (an executed witness for contradictions, sealed round-trip tests) is what the second evaluation measured. The remaining gap, a test writer that sometimes cannot seal a valid suite, is in the report too.

What's next

A check on every agent pull request (planned, not built), outside pilots on real repositories (PILOT.md), and a better sealing loop.

Feedback on Nebius Token Factory, Sandboxes and NVIDIA models

  1. Sandboxes: spawn idempotency. On 21 Oct, 6 of the instance spawns of our evaluation (POST /sandboxes/v1/instances) were answered 500 Internal Server Error after the request was sent, while status.nebius.com showed every service operational. The request carries no client-supplied id, so a client cannot know whether the instance was created; we had to list GET /operations?kind=instance&since=&until= and match image UUIDs (no matching operation was found in bounded read-only checks, including a follow-up at least one hour later). An idempotency key or client operation id on spawn, and a documented meaning of a 5xx (created or not), would make safe retries possible.
  2. Sandboxes: late creation. On 17 Oct a spawn whose request timed out on our side was created 101 s, and another 262 s, after it was sent (it ran after its run had ended). A client timeout does not cancel a queued spawn; documenting that, or a cancel-by-client-id, would help.
  3. Token Factory docs: structured output. The structured-output examples pass the raw schema under json_schema; the live API answered 422 (response_format.json_schema.name missing) until the wrapped {"name", "schema", "strict": true} form was sent.
  4. Nemotron 3 Super reasoning budget. With reasoning on, Super spent the whole completion budget on reasoning; switching reasoning off lowered implementation quality; reasoning_effort bounded it, and the sealer still needed low effort to stay inside 16,000 tokens.
  5. Sandboxes configuration names. The SDK reads NEBIUS_PROJECT_ID while the CLI reads NEBIUS_AI_PROJECT.

Built with

Python, NVIDIA Nemotron 3 (Ultra, Super) on Nebius Token Factory, Nebius Token Factory Sandboxes, Tavily, FastAPI, SQLite, nginx.

Links

Project history

New project: designed on 14 Oct 2026 and built from 15 Oct 2026, inside the submission period (26 Aug to 30 Oct 2026).

Built With

Share this project:

Updates

Submission history