EffectGate

Making AI agents prove their code works before it ships.

Comment 1

EffectGate

Making coding Agents prove their fixes before Git accepts them.

Executive summary

Problem and inspiration

Coding Agents can inspect a repository, edit files, run tools and report that a repair is complete. The dangerous gap comes afterward: a plausible patch and a green test do not establish that the test would have caught the original bug, that the patch solves more than one visible example, or that unrelated behavior still works.

This is a documented problem in automated program repair. SWE-bench evaluates patches against fixed repository states and issue-specific tests. Qi et al. showed that weak test suites can accept plausible but incorrect generated patches. A 2025 analysis of SWE-bench Verified found accepted patches that failed deeper developer or differential tests.

The lesson is narrow but important: candidate-only success is not enough evidence to authorize a protected effect.

Solution

EffectGate is lightweight release-control middleware between a coding Agent and Git.

Before the Agent starts a task, EffectGate saves the complete starting workspace and records the request. The Agent then works normally inside a restricted Docker runtime. When code changes are ready, the interface unlocks Check & submit.

EffectGate accepts two independent sources for its acceptance test. A repository may register an incident reproducer before the coding Agent starts. If none exists, Check & submit starts a separate test-author context that receives only the saved starting code and task, never the proposed patch. In both paths, EffectGate proves that the test reaches the intended assertion failure twice on the original code, then locks it outside the coding workspace.

EffectGate reconstructs the exact original and proposed states and allows Git submission only when:

  1. The locked test fails for the expected reason twice before the fix.
  2. The same test passes twice after the fix.
  3. Any platform-owned hidden check confirms the broader behavior.
  4. The existing project test suite remains green.
  5. The complete workspace delta contains no forbidden secret, repository, dependency, build or existing-test change.
  6. The base revision, frozen files, evidence digest and one-use approval still match at the Git boundary.

If evidence is insufficient, Git stays unchanged. EffectGate gives one typed failure to the same Agent thread and permits one bounded correction. If every check passes, it issues a short-lived Ed25519 approval bound to one exact change and one protected Git action.

Exact claim

EffectGate does not claim that tests prove universal correctness. It establishes that a locked test, plus any platform-owned hidden check, reproducibly distinguishes one exact proposed change from one exact starting state under a declared contract. It then ensures that Git receives the same files that produced that evidence.

System architecture and tech stack

Data flow

Developer
  |
  v
React coding workspace
  |
  v
Fastify request boundary
  |
  v
AgentService middleware
  |-- save starting workspace and task
  |-- run coding Agent in restricted Docker
  |-- expose Check & submit only after changes exist
  |
  v
EffectGate control plane
  |-- use a pre-task incident test, or let a separate test author see starting code only
  |-- validate test twice on original code
  |-- freeze original, proposed change and locked test
  |-- run original, proposed, hidden and project checks
  |-- stream evidence events to the interface
  |
  +-- insufficient evidence --> Git unchanged --> one Agent correction
  |
  `-- sufficient evidence --> one-use Ed25519 approval
                                  |
                                  v
                         Protected Git provider
                                  |
                                  v
                         Draft ref and exact commit

Trust boundaries

Boundary Responsibility
Coding Agent runtime Can edit only its workspace. It has no Git client, protected remote, signing key or provider credentials.
Acceptance-test source Uses a pre-task incident reproducer when available. Otherwise, a separate test-author runtime sees the frozen starting project and task, not the proposed change.
EffectGate control plane Owns checkpoints, immutable artifacts, policy, evidence events and the signing key.
Restricted verifier Executes selected tests in fresh, networkless Docker containers with read-only inputs and resource limits.
Protected Git provider Revalidates the signature, base, evidence and candidate tree before creating a draft.

Tech stack

Layer Technology Purpose
Interface React 19, TypeScript, Vite Agent conversation, live repository, release state and evidence drawer.
Middleware Node.js 22, Fastify 5, Zod Request boundary, Agent lifecycle, checkpointing and validated control APIs.
Coding Agent Codex CLI, ModelArk Responses API, Docker Real multi-turn coding with restricted outbound access.
Verification Docker, Vitest, TypeScript Repeatable target, hidden and project-suite execution.
Integrity Canonical JSON, domain-separated SHA-256 Stable identities for contracts, files, results and provider actions.
Authorization Ed25519 Short-lived, single-use approval for one exact artifact and Git action.
Protected effect Local bare Git repository Inspectable draft ref and exact commit after final provider validation.

Every verifier run uses a fresh container with no network, a read-only root and artifact mount, a non-root user, dropped capabilities, no-new-privileges, resource limits, a timeout and confirmed cleanup.

End-to-end workflow and demo scenario

1. A real repository and incident

Demo mode opens one preloaded TypeScript repository called Workspace Files API. The service reads files requested by coding tools from a configured workspace. Its resolver currently allows ../../outside.txt to escape that root.

The prepared incident asks the coding Agent to reproduce a deliberately narrow emergency hotfix that blocks only the reported literal path. This is disclosed as a deterministic adversarial fixture, not presented as a random model failure.

2. The Agent changes code

The Agent receives the task through the ordinary chat API and edits the repository inside Docker. EffectGate has already saved the vulnerable starting version. While the Agent is running, submission is unavailable. When the Agent finishes and a complete delta exists, the status changes to Ready to check and Check & submit becomes available.

3. EffectGate stops the narrow fix

The user selects Check & submit. This incident already has a platform-owned test for the reported exploit, locked before the coding Agent began. EffectGate validates that test against the starting code, freezes the proposed files, then runs every check in isolated containers.

Locked test on original:       EXPECTED_ASSERTION_FAILURE, twice
Locked test on narrow fix:     PASS, twice
Hidden traversal check:        FAIL before, FAIL after
Project test suite:            PASS
Decision:                      BLOCKED_ORACLE_FAILURE
Git drafts created:            0

The visible exploit is blocked, but another traversal path still escapes. The interface says A hidden case still fails and confirms Git: No change sent.

4. Evidence improves the Agent

The user selects Improve with Agent. EffectGate sends the typed failure to the same coding thread without exposing or unlocking the hidden test. The Agent replaces the literal check with a general containment rule. A new immutable child attempt reruns every phase.

Locked test on original:       EXPECTED_ASSERTION_FAILURE, twice
Locked test on corrected fix:  PASS, twice
Hidden traversal check:        FAIL before, PASS after
Project test suite:            PASS
Decision:                      ELIGIBLE

The protected provider rechecks the signed approval, expiry, one-use state, base revision, evidence digest and candidate tree. Only then does it create refs/effectgate/drafts/<attempt-id> and the matching commit.

Three design choices

1. Keep acceptance evidence independent of the proposed answer

Known incident reproducers are registered before coding. For ordinary tasks without one, the test-author context receives only the pre-task checkpoint and intent. Both paths prevent the proposed patch from directly shaping its own acceptance test.

2. Bind evidence to the effect it authorizes

SHA-256 digests identify the exact contract, files, results and Git action. Ed25519 approval is valid only for that frozen combination. The provider validates every bound value again at the final boundary. This applies the artifact-integrity principle demonstrated by in-toto, although EffectGate is not an in-toto implementation.

3. Make recovery useful and bounded

EffectGate classifies weak tests, candidate failures, hidden-check failures, suite regressions, forbidden files, instability and infrastructure errors. One evidence-guided correction demonstrates a feedback loop without allowing unlimited attempts to adapt to the verifier.

Key innovations

  • Counterfactual release evidence: fail-before and pass-after behavior directly controls whether Git can change.
  • Independent test provenance: the acceptance test is either registered before coding or authored from only the pre-task checkpoint and intent.
  • Evidence-bound authority: a passing result creates narrow capability for one exact artifact, not a reusable approval.
  • Complete-delta enforcement: EffectGate inventories the workspace instead of trusting an Agent summary or hand-selected diff.
  • Visible enforcement: the interface streams the real check stages, reason for rejection, cleanup and final Git outcome.
  • Evidence feedback: a typed failure returns to the same Agent once, and the corrected child attempt must satisfy the unchanged checks.

Verification

Automated tests cover checkpointing, generated-test validation, hidden-check failures, candidate failures, suite regressions, missing targets, syntax and import failures, inconsistent results, forbidden files, artifact substitution, stale bases, invalid or reused approvals, provider recovery, bounded correction, redaction, real Docker isolation and cleanup.

npm run check
npm run demo:verify
npm run evaluate:verify

The evaluation suite is conformance evidence. It is not presented as a real-world Agent benchmark.

Future plans

  1. Evaluate independently judged repairs from real repositories and publish false acceptance, false rejection, correction success and latency.
  2. Add GitHub and GitLab providers, more test adapters and hardened microVM verification.

References

  1. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
  2. Qi et al. An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems
  3. Wang, Pradel and Liu. Are "Solved Issues" in SWE-bench Really Solved Correctly?
  4. Torres-Arias et al. in-toto: Providing Farm-to-Table Guarantees for Bits and Bytes

Built With

Share this project:

Updates

Submission history