Shadow Commit

Logo
Outro
Website

Inspiration

In the past year, coding agents from four different companies shipped security holes with the same shape. Two were in Anthropic's Claude Code, one in OpenAI's Codex was rated 9.8 out of 10, one was in GitHub Copilot, and Pillar Security found a whole series more. In every case the agent wrote something, a trusted part of the system ran it later, and the check in the middle never saw it.

We then recorded our own version of the problem. We gave an agent a hello-world task, and it reported "All steps complete, Jest tests passed, 2/2." The journal for the same turn recorded nine commands run, six failed, and the test command itself exiting with code 1. The summary an agent gives you and the effects it actually produces are two different things, and only one of them is true.

Most platforms respond by screening commands before they run. A classifier reads the words and guesses what will happen. But a command can be spelled in a way the classifier has never seen, and a destination can be built while the code runs, after the reading is done. The incidents above are what happens when the guess is wrong. We wanted a control that does not guess.

What it does

Shadow Commit is middleware that makes every agent turn a transaction. A turn moves through five steps:

Seal. The workspace is copied and the copy is verified. The agent's memory is sealed with it. A per-turn private network comes up with no route to the internet, the egress broker is the only way out, and the model key is swapped for a one-turn token.

Run. The agent works normally on the copy. It is never told it is sandboxed.

Capture. The sealed copy is diffed against what the agent left behind. Every created, changed, or deleted file, plus every held network request, becomes the effect set. The agent does not get to write this list.

Judge. Eighteen rules read every effect. Nothing stops early, and the worst decision wins. A rule reads what a command did to the files, not how it was spelled.

Settle. A clean turn commits, verified byte by byte against hashes recorded at judgment time. A dangerous turn is discarded and the real workspace is untouched. A doubtful turn is held for a person, shown as a plain list of changes.

Every step appends to a tamper-evident journal: each record carries the SHA-256 hash of the record before it, every record is signed with an HMAC, and signed checkpoints pin the chain at intervals. Flip one byte of one record and verification refuses on three independent layers.

How we built it

The platform is the hackathon starter kit; the middleware is one seam. createRunner() composes a decorator around the kit's own AgentRunner interface, so every agent gets the transaction with no configuration and the kit's UI, API, and playground keep working untouched. The server is TypeScript on Fastify with Zod at the boundaries, the UI is React and Vite, isolation is Docker with network create --internal, and the broker allows destinations only on an exact host and port match.

The transactional idea has a published paper: Cordon (arXiv 2606.17573, June 2026) introduced transactions for tool-using agents ten weeks before this project and validated them on 45 risk cases. We built in three directions the paper does not cover: it is middleware in one decorator rather than a 14.4 thousand line custom runtime, it rolls the agent's memory back along with the files, and it publishes the costs of the control, not only the containment rate.

Challenges we ran into

The hardest problems were measurement, not code.

Our first containment numbers were computed on the same corpus the policy was tuned against, and the discard tier had been calibrated against a set that could not charge it for a false positive. Every arm of that setup looked better than it was. We rebuilt grading around work nobody here generated: real commits from open-source repositories, written by developers who never saw the policy.

Effect capture had to be correct against restoration tricks. A file whose stats are restored after an edit looks untouched; only a content hash recorded at seal time catches it.

Early tests asserted against mocks and quietly encoded the mock's behavior as the contract. Several checks failed while the product was correct. We rewrote them to assert what must hold for the real runtime.

Accomplishments that we're proud of

Measurement	Result
Attacks contained, 3,251 policy-decidable scenarios	96.5% (missed 3.5%, down from 58.5% before these controls)
Honest work wrongly discarded	0.5%, about one turn in two hundred
Honest turns routed to a person for one click	18.0%, with the per-rule breakdown published; one rule carries 712 of the 902 holds
Blind check: 998 real bug-fixing commits by outside developers	3 held back, 0.30%
Replay of 19,102 real commits	73.01% committed without a person, 26.19% held, 0.80% destroyed
Test suite shipped with the kit	2,385 server tests and 172 web tests, all green

Every number above regenerates from a fresh clone. npm run corpus rebuilds the 8,285-scenario corpus, 3,280 attacks in 15 families plus 5,005 benign turns, and replays it byte-identically. Three gates then re-derive each published figure from the measurement artifacts, check 29 pinned constants, and count the rule tiers, so the prose cannot drift from the measurements without a gate failing.

When the AgentDyn survey re-tested ten published defenses on held-out data, honest-task utility retained ranged from 0 percent for CaMeL and ProtectAI through 12.5 percent for Progent to 56.3 percent for DRIFT, the best measured. On our own held-out measure, 19,102 real commits, Shadow Commit passes 73.01 percent untouched. We also publish a number the field does not report at all: the fraction of honest work irrecoverably destroyed, 0.80 percent. The comparison and its caveats are in research/benchmarking/POSITIONING.md.

The recorded demo drives one agent through five turns, one for each settlement outcome. In the attack turn, "install a release helper" tried to reach a cloud credential endpoint, contact a staged exfiltration server, plant a git hook, and delete a protected customer file, then reported success. The broker refused the destinations, the policy discarded the turn, and the workspace fingerprints before and after are identical. Everything replays from committed files with a mock model and no API key.

What we learned

Do not read the agent's summary. Count the effects. An effect does not have to be guessed.

A number measured on data the thing was built from is a floor, not a result. Publish the blind one beside it.

The verdict tier matters as much as the detection logic. Most of our early failures were fixed by changing what the verdict tier did. The strongest evidence is a counterfactual replay in the repository: every destruction of honest work could become a recoverable hold with zero loss of attack containment, at the cost of flooding the review queue under sustained attack, where turns needing a person rise from 5.9 percent to 95.5 percent. Both sides of that trade are published.

What's next for Shadow Commit

Inspection inside encrypted egress. Today an allowed tunnel is checked by destination but not content, and our weakest attack family, secrets hidden in encoded blobs at a 10% miss rate, is the clearest reason to build it.

Mid-turn cancellation on camera. The kill switch exists in code and settles the turn cleanly; the next step is demonstrating it live.

Growing past one machine. The pieces to swap for a hardened multi-host deployment are written down in the repository.

The trust boundary: an untrusted zone for everything one agent turn can reach, an egress broker as the only route out, and a trusted zone that decides

The attack turn's effect set read by all eighteen rules, worst decision wins

The measured results: 96.5 percent of decidable attacks contained, 0.5 percent of ordinary work wrongly discarded, 0.30 percent held on the blind check