Counterfactual Release Lab

GitLab Duo picks failure worlds. Deterministic evidence decides whether code is safe to ship.

Comment

AI chooses what to challenge. Deterministic evidence decides whether a candidate survives.

Open the live lab · Watch the 2:19 demo

Inspiration

A green build can hide an assumption that fails under pressure: a coordinator's authority expires, a handoff loses packets, a provider call is interrupted, or test evidence no longer matches the source.

Counterfactual Release Lab gives an agent the job of selecting failure worlds and gives reproducible tests the final verdict. It adds new post-code automation around the existing MIT-licensed WayLatch project for Path B: Bring Your Own. WayLatch is the application under test; Counterfactual Release Lab is the verification and evidence workflow.

Try the live demo

No account or credentials are required. Free hosting may take about a minute to wake.

  1. Choose Run safe candidate. The service starts real verifier processes against a bundled synthetic Git fixture and checks all five registered worlds.
  2. Choose Run expiry regression. A separate fixture removes one expiry predicate. The verifier returns BLOCK, preserving both original expiry failure reasons and the actual failed assertions.
  3. Edit a failure reason in the execution receipt, then choose Verify edited receipt. The digest mismatch must block verification.
  4. Choose Restore execution receipt, download the evidence, and refresh to recover the same run.

Fresh public runs use deterministic-baseline planning. Authentic historical GitLab Duo selections are labeled separately and link to their original reports.

What GitLab Duo does

A registered GitLab Duo Agent Platform flow selects scenario identifiers and reasons from a fixed catalogue:

  • Offline persistence: data survives storage connections.
  • QR resilience: loss, corruption, duplication and replay.
  • Coordinator-session expiry: expired authority must be rejected.
  • Payment interruption: interrupted provider calls, using controlled test doubles.
  • Evidence freshness: source, revision, plan and report must match.

Scenario generation is bounded selection from this catalogue. Model text cannot supply arbitrary executable commands or approve its own release.

Two authentic recorded sessions demonstrate the agent's role:

  • Session 9739808: Duo selected evidence freshness; the reviewed safe candidate passed.
  • Session 9739811: Duo selected coordinator-session expiry, evidence freshness and payment interruption; behavioral assertions rejected the defective candidate. Both primary expiry reasons survived a later finalization error.

How it works

  1. Select the challenge. Duo selects registered worlds and explains their relevance.
  2. Run the verifier. A pinned, reviewed verifier executes registered behavioral tests against the candidate.
  3. Bind the evidence. Candidate revision, source digest, selection plan and report hashes must agree. Missing, stale, incomplete or failing evidence blocks progress.
  4. Preserve the decision. Original failure reasons remain visible in the resulting receipt.

The public lab uses a small Express broker and a dedicated HTML/JavaScript interface. It accepts only two fixed fixture identifiers and launches the existing CLI with a whitelisted environment. It does not accept uploaded repositories, supplied commands or production credentials.

Fixture histories are built from the reviewed image source. Their commit identities are explicitly synthetic and distinct from real GitLab candidates. The defective fixture runs only inside fixed tests; it is never served as the public application and is not MR !6.

Codex/ChatGPT assisted implementation and cloud QA. GitLab Duo performed the recorded scenario-selection decisions.

Verification

Anonymous public acceptance on 21 October 2026 confirmed:

  • Safe candidate: all five worlds passed.
  • Expiry regression: blocked with exactly the two expected original failure reasons.
  • Receipt integrity: tampering and safe-result replay were rejected; restoration and evidence download worked.
  • Browser behavior: refresh recovery and 320, 390, 768 and 1440-pixel layouts passed, with no browser errors or failed asset requests.
  • Published assets: all five served assets matched the reviewed source byte for byte.
  • Final Pages Black Box: 28 checks passed.

Backend commit 7611a166 passed pipeline 2970079800: 248 Node tests, six Python tests, and both real fixture executions in an isolated container. The documentation and Pages publication passed pipeline 2970103356.

The container gate runs as an unprivileged user with no external network, no published ports, a 512 MiB memory limit and 0.5 CPU. Public HTTPS acceptance is checked separately from CI.

Limits and safeguards

  • Supervised verification: both historical and fresh decisions retain releaseAllowed: false and NOT_DEPLOYED. Passing scenarios do not grant production release authority. A full hands-off code-to-production loop is not demonstrated.
  • Bounded tests: fake IndexedDB, raw QR text, provider doubles and a simulated expiry clock. No physical camera test, real payment, arbitrary remote candidate or fresh public Duo invocation is claimed.
  • Disclosed timing allowance: bundled fixtures allow 15 seconds per test on shared Free CPU, compared with the unchanged production default of five seconds. Both fixtures use the same assertion code and process deadlines. Each receipt records the adaptation.
  • Receipt consistency: SHA-256 checks compare a receipt with its execution; they are not an independent digital signature.
  • Free hosting: one active execution, 24 starts per hour and eight retained receipts. Results expire after ten minutes. Idle hosting can sleep; restarts clear jobs, and exhausted quotas can suspend service.
  • Protected resources: the Render service is isolated from the original WayLatch workspace. No paid resource was activated. Automatic Duo trigger 3072145 remains disabled; unsafe MR !6 remains unmerged and undeployed.

Source and reproduction

Use the isolated deployment branch for the live lab source. The default branch preserves the earlier merged engineering checkpoint. Build .gitlab/deployment/Dockerfile and configure a canonical APP_ORIGIN; no provider credentials or model downloads are needed.

Demonstration video

Watch Counterfactual Release Lab on YouTube

The prepared 2:19 film covers fresh hosted verifier execution, genuine safe and blocked outcomes, receipt tamper/restore checks, completed CI and clearly labeled historical Duo decisions. Actual waits are condensed; public runs use deterministic-baseline planning.

Built With

Share this project:

Updates

Submission history