RewardSeal

RewardSeal checks whether a coding-agent training task can be passed without solving it, and records the evidence.

The problem. Coding agents are trained and scored on tasks with hidden tests. If the answer is lying in the task's own git history, an agent can copy it, pass the tests and learn nothing. The people who build these tasks have no cheap way to look for that.

What it does. RewardSeal (1) scans a task for exposed fixes, (2) lets a deliberately dishonest attacker, NVIDIA Nemotron 3 Super served through Nebius Token Factory and running its shell tool inside Token Factory Sandboxes, try to pass the task without solving it, (3) applies a rule-based patch such as strip-future-git, (4) re-checks that the correct patch still passes and an empty patch still fails, and (5) re-attacks and writes a certificate. Gates and labels are computed in code, not by a model.

What we found (all recorded, with caveats on the site).

• In 12 of 13 audited environments, every counted line of the gold fix also appeared in a future commit (line overlap; a selected sample of the 18 lowest-scoring tasks, 5 excluded as environment-broken; not proof the whole fix is retrievable). After the patch, no future commits remained in the task repo and both gates held in 11 of those 12.
• On one task (CycloneDX 348) the attacker ran git show on the fix commit and then wrote a matching line that passed the hidden tests. That is a history-use pattern in 1 of 9 recorded before-rounds; use of the output is inferred from command order. A replay with no model, in the recorded 348 image (digest not retained), matched the recorded output hashes 3 of 3.
• A separate random sample: 7 of 9 audited tasks had at least half the counted lines in future commits, and 5 of those 7 were closed.
• Tavily is called at runtime for a harness recon search and for a web-exposure audit (0 of 13, and 0 of 9 on a separate sample, for one tool on one day: findability, not proof the web lacks the fix).

Honest limits. "Certified (k=1)" means one completed attack failed under the recorded conditions, with exposure removed and the gates checked by code. It is not a security guarantee. The hardened re-attack ran with different recon from the original, so a same-conditions re-attack is not shown. Samples are small, one attacker and prompt family, no real users or measured time savings yet.

Try it. Open the report (no login): https://manishwvn.github.io/rewardseal/ , press Verify on https://manishwvn.github.io/rewardseal/verify.html to recompute the evidence hashes in your browser. Offline commands are in the README.

Built With

• github
• nebius
• nemotron
• nvidia
• pytest
• python
• sandboxes
• tavily
• token-factory
• uv