FORK — Evidence Firewall for AI Code Agents

AI wrote the patch. Human approval stayed bound to the reviewed bytes.

Comment

Elevator pitch

FORK lets Codex investigate a real repository, author a previously absent regression test, and propose a repair—but it will apply nothing until the exact red test is locked and a human separately approves the exact production patch hash. It then reruns the unchanged test and ordinary suite and emits one reviewable evidence record.

Inspiration

AI coding agents can write both the answer and the exam. That makes them productive, but it also creates a credibility gap: after seeing a failure, the same agent could weaken the test, change a fixture, broaden the patch, or apply bytes different from the diff a person reviewed. A final green check does not explain which claim stayed constant.

Ghost Deploy made that problem concrete. In our executable incident, a release commits but its acknowledgement is lost. The release later fails health checks, a human rolls back and revokes approval, and a delayed retry wakes with stale authority and redeploys the rejected build. Every existing test is green, yet production violates a human decision.

We did not want another agent that merely says, “I fixed it.” We wanted a narrow permission boundary that lets models create while preventing them from changing the evidence or authority used to judge their own work.

What it does

FORK runs inside an actual Git repository as a Codex plugin and deterministic local verifier.

  1. A session records the base commit, clean working-tree scope, declared source identities, production-path allowlist, and ordinary test commands.
  2. Codex inspects the implementation and authors a new focused regression test after initialization, before any production edit.
  3. FORK accepts the lock only after the exact controlled node:test command genuinely fails with the expected assertion marker. It binds the test bytes, command, runtime, source state, and red output.
  4. Codex writes an unapplied production repair. FORK rejects candidates that touch the locked test, session evidence, or paths outside the declared production scope.
  5. FORK displays and hashes the exact accepted diff. A human reviews it and separately approves that SHA-256 identity.
  6. FORK recomputes the hash, applies only those approved bytes, reruns the unchanged locked test, and then runs the ordinary suite.
  7. A successful session emits fork-repair-evidence@2, joining the base state, red lock, proposal, approval, application, unchanged green run, and ordinary results.

The hosted viewer validates and renders that artifact in the browser. It accepts a valid record and rejects a one-byte-altered record. Selected JSON stays in the browser tab; the website cannot read, modify, or approve changes in a local repository.

The difference from a conventional autofix agent is simple: Codex can author the test and patch, but FORK owns the transitions between claim, permission, and result.

The live demo

The recorded demo starts with the executable Ghost Deploy failure in a real repository—not a prepared “broken/fixed” web animation. Ordinary tests pass while a stale retry redeploys a release after rollback and approval revocation.

Codex inspects the repository and creates a regression test that did not exist at session start. FORK observes and locks it red. We deliberately submit a candidate that changes the locked test; FORK rejects it before approval. Codex then produces a legitimate production-only diff, which FORK validates and hashes without applying.

Before human approval, GPT-5.6 Sol reviews only the bounded task, generated-test expectation, red excerpt, changed-file list, exact unapplied diff, and patch hash. Its strict structured advisory contains riskSignals, assumptions, reviewQuestions, recommendedChecks, and confidence. That output has no authority to decide pass or fail, approve or reject a diff, apply bytes, or unlock a gate.

A person reviews the complete diff and separately approves its exact hash, and FORK applies only that patch. The same test bytes turn green, the ordinary suite stays green, and the generated evidence artifact passes the viewer. A copy altered by one character is rejected.

The submitted video uses clearly disclosed AI-generated English narration created with OpenAI gpt-4o-mini-tts and the Cedar voice. It contains no background music or stock footage. The public 2:27 demo is available at https://youtu.be/WFx6y4nsU9w.

The submitted Ghost Deploy recording session reached FORK's verified phase. It binds patch hash sha256:9441c27c4725e3c00c3db24519de63a0c86b52613e77a29298eb703013454482, unchanged test hash sha256:75acaa6d035ba76c21829c4377758c62565bd4957df507874038f5a2e4143298, and final Git diff hash sha256:6efca43d0291beadabf39e963c6e9cec513478433cc264f7c21f15b2b46b267b in its generated evidence.

How we built it

The repair engine is a staged TypeScript command-line workflow: init, lock-test, propose, approve, apply, and verify. Each stage has a narrow input contract and can fail independently.

The red-lock adapter currently permits only the same absolute Node runtime executing FORK and the exact argv shape node --test <locked-test>. The required assertion marker must exist in both the new test source and observed red output. This prevents a wrapper from accepting a filename while silently running something else.

Patch handling uses strict Git unified diffs, conservative repository-relative path parsing, protected-path rules, git apply --check, SHA-256 byte identities, and post-apply source and diff hashes. Approval names one exact patch identity. Verification fails if the locked test, command, protected paths, source baseline, proposed bytes, applied bytes, focused result, or ordinary suite no longer matches the recorded workflow.

The web experience uses React 19, TypeScript, vinext, Vite, and OpenAI Sites. Its local browser validator enforces the evidence@2 structure, content hashes, path bindings, exact command shape, patch identity, approval match, source transitions, final diff identity, and unchanged-test condition before rendering a record.

We also built a deterministic Ghost Deploy schedule explorer covering 1,440 bounded event orders. It is a secondary reproducibility tool for understanding the timing bug, not evidence that a model performed the recorded repair.

How we used Codex

Codex was both our implementation partner and the constrained repair author demonstrated by the product. During Build Week, we used it to inspect and evolve the repository, implement the staged session engine and browser evidence path, write adversarial tests, package the plugin skills, diagnose integration failures, and run repeated build and verification passes. The primary linked Codex session contains the majority of the core development work and decisions.

Inside the final product demo, Codex operates on the actual repository. It investigates the incident, authors the new regression test, and creates the unapplied production-only repair. FORK does not trust a model-generated success statement; it converts Codex's work into independently checkable gates and stops for a separate human approval.

How we used GPT-5.6 Sol

GPT-5.6 Sol is called through the OpenAI Responses API with strict JSON Schema output. It receives only bounded task, generated-test expectation, red excerpt, changed-file list, exact unapplied diff, and patch hash. It returns riskSignals, assumptions, reviewQuestions, recommendedChecks, and a confidence level about context sufficiency.

Its role is advisory by design. GPT-5.6 Sol cannot decide the regression result, approve or reject a patch, apply bytes, modify evidence, or mark the session complete. Deterministic repository commands and exact identities control those transitions. The UI exposes the model name and liveModelCall provenance; if the live request is unavailable or invalid, the product visibly labels its versioned fallback liveModelCall: false instead of presenting it as a live response.

Challenges

The hardest challenge was making the demonstration credible. A prewritten broken candidate and fixed candidate can teach a timing failure, but they cannot prove an AI repair happened. We moved the primary proof into a real Git workflow where the new test is visibly absent at initialization, created by Codex, observed red before production changes, and preserved through the green result.

The second challenge was keeping one claim intact across a mutable working tree. Filenames were not enough. Test bytes, command arguments, runtime, base revision, source hashes, patch paths, proposal bytes, approval identity, applied diff, and output all needed explicit bindings and negative tests.

The third challenge was separating intelligence from authority. GPT-5.6 Sol is useful at surfacing risks and questions, and Codex is useful at authoring tests and repairs. Neither should silently decide that its own work is acceptable.

Accomplishments we are proud of

  • A real repository workflow from Codex-authored red test to human-approved, unchanged-test green evidence
  • A completed Ghost Deploy session with a verified patch, unchanged-test, and final Git diff identity chain
  • A visible rejection path for patches that alter the locked test or protected evidence
  • Separate proposal, approval, application, and final-diff identities instead of one ambiguous “fixed” state
  • A browser viewer that accepts the recorded artifact and rejects a one-byte tamper without uploading repository data
  • Adversarial coverage for changed tests, stale hashes, protected paths, invalid runner shapes, baseline drift, and failed verification commands
  • An already-bundled Codex plugin and deterministic sample path that judges can exercise without compiling the project
  • A bounded GPT-5.6 Sol advisory whose structured output is useful without being treated as authority

What we learned

The strongest AI workflow is neither “trust the agent” nor “ban the agent.” It is controlled authorship with independent authority. Models should be free to investigate and create, while consequential transitions remain bound to deterministic evidence and an explicit human decision.

We also learned that a benchmark, a demo, and proof are different things. A deterministic schedule explorer can reproduce a class of failure. The real product proof is the continuous repository chain: new test, observed red result, immutable lock, exact reviewed diff, distinct approval, unchanged green result, and reviewable artifact.

How judges can test it

The repository includes the complete fork Codex plugin and its already-bundled .mjs executables. No TypeScript build is required to inspect the workflow or run the included verifier. Clone the source repository from the final links, then run these commands at its root:

codex plugin marketplace add .
codex plugin add fork@fork-build-week
node plugins/fork/skills/fork-repair/scripts/fork-session.mjs --help

The marketplace's explicit identity is fork-build-week, avoiding a collision with a judge's personal marketplace. Restart Codex desktop and start a new task so the installed skills load. Open the supplied demo repository and invoke $fork-repair. The included judge quickstart provides the exact task, config, paths, and commands. Codex CLI users can open /plugins to inspect the same installed plugin.

The submitted and demonstrated platform is Codex desktop on Windows 11 with Git and Node.js 22.13 or newer. The local verifier is written with cross-platform Node and Git primitives, but macOS and Linux are portability targets rather than platforms claimed as validated in this submission. The plugin is not presented as an IDE-extension or mobile experience.

Judges can evaluate FORK without rebuilding in three ways:

  1. Open https://fork-green-lie.diskotek1071.chatgpt.site/, load its sanitized copy of the verified Ghost Deploy artifact, then choose the one-byte tamper action to see validation fail.
  2. Run the repository's already-bundled plugin executable against the supplied demo repository by following the judge quickstart.
  3. Watch the link-accessible, narrated, under-three-minute YouTube video for the real on-screen GPT-5.6 Sol advisory and permissioned Codex repair flow.

The hosted experience is public and free of charge. The private repository will be available free of charge to authorized judges once the official invitations are accepted; both required invitations have been sent and remain pending acceptance at the time of writing. Repository materials are proprietary and evaluation-only; no open-source or reuse license is granted. See LICENSE.

Technologies used

  • OpenAI Codex and the Codex plugin/skill system
  • GPT-5.6 Sol through the OpenAI Responses API with strict structured output
  • OpenAI gpt-4o-mini-tts with the Cedar voice for disclosed English narration
  • OpenAI Sites
  • TypeScript, Node.js, React 19, vinext, Vite, and Git
  • SHA-256 content identities and strict unified-diff validation

Third-party software and synthetic data

FORK uses the OpenAI Responses API, Codex, OpenAI Sites, React, Node.js, vinext, Vite, and Git. The Ghost Deploy repository, release names, approval events, incident timeline, and evidence artifact contain entirely synthetic local demo data. No production deployment system, customer repository, payment account, or personal dataset is connected to the demonstration. The project includes no third-party music, stock footage, or externally owned benchmark data.

Honest limitations

FORK is not formal verification and does not prove that arbitrary software is safe. A genuine red-to-green regression test can still be incomplete or encode the wrong requirement. The production scope is declared configuration, not semantic understanding of every repository.

The current approval record binds exact patch bytes but does not cryptographically attest the approver's identity. Evidence is stored in a maintainer-controlled environment rather than an external transparency log, so it is not described as tamper-proof. The test generator field is a local workflow claim, not model-identity attestation.

The controlled red gate currently supports Node's built-in node:test runner only. Ordinary post-repair commands remain configurable, but Jest, Vitest, pytest, signed runner adapters, CI attestation, and remote approval services are future work. The current patch parser intentionally accepts a conservative ASCII path alphabet and rejects ambiguous Git quoting.

External executables, ignored dependency caches, the operating system, and the wider software supply chain are outside the current lock. The hosted viewer cannot access or edit a local repository. The separate 1,440-schedule explorer covers a finite declared model, and its bundled candidates are reference fixtures rather than evidence of a live AI repair.

What's next

Next we will add signed or remotely attested approvals, CI verification of evidence@2, pull-request annotations, adapters for additional test runners, repository policy presets, and transparency-log publication. We also plan more executable incidents for payment retries, job queues, access revocation, and migration rollback.

Built With

Share this project:

Updates