Stale

Stale: re-verify what you already shipped
MR pipeline: tests, build, SAST, secret detection, container scanning, re-proof, verify-mr
Every DevSecOps stage maps to a real job
What changed in the world (live feeds and database stats) and the last rollout
The control loop opens an issue with an evidence table and a machine block
The agent's fix: Node image bump plus the distrusted root's key excluded
Live status page: each claim's path from issue to Duo MR, release and mint
Four real incidents, one pattern: a safety verdict nobody re-checked
A claim is a file: assertion, deterministic rules, and the only paths a fix may touch
The hands-off loop: the world changes → control loop → GitLab Duo flow → CI proof → canary release
The GitLab Duo stale-remediator session: reads the issue and playbook, opens one MR
Shadow migration on 600,000 rows under write load: CREATE INDEX CONCURRENTLY, no blocking lock
Canary probes: trusted partner 200, distrusted partner 502 with a certificate error
Release gate DEFER: the first fix waits while the other claim's fix is in flight
Mid-loop: both claims REPROVING, evidence failing, release running in CI

Inspiration

On October 20 Mozilla pulled emSign ECC Root CA – G3 out of its root store after weak eMudhra CA keys were found, and a day later Node.js shipped v24.22.1 with the updated roots. Any Node service that had been marked "safe to ship" the week before was still trusting that root. Its pipeline had no reason to run again, because nothing in its repo had changed.

Once we looked for that pattern we kept finding it. Railway's October 2025 outage came from an index migration that was fine when the table was small and locked it for about 30 minutes once it held around a billion rows. In March 2026 a pinned axios version that everyone trusted turned malicious overnight. CrowdStrike's Channel File 291 shipped content whose compatibility was assumed, never re-checked. Each time, a safety verdict expired and nobody re-checked it.

CI only re-checks things when code changes. We wanted a change in the outside world to count as a trigger too.

What it does

Stale treats every "safe to ship" verdict as a cached value that has evidence behind it.

A claim is a YAML file in the repo. It says what must stay true, where the evidence comes from, which deterministic rules decide that the claim has gone stale, which files a fix is allowed to touch, and how to prove the fix worked.

Every 15 minutes a scheduled GitLab pipeline collects evidence from the live Cloud Run service, from public feeds (Node releases and root-program distrusts), and from live Postgres statistics. If a rule fires, it opens an issue containing the exact facts a fix needs and assigns it to our GitLab Duo custom flow, stale-remediator. The flow reads the issue and the claim's playbook and opens one merge request.

The MR pipeline then checks policy (allowed paths only, who made the commits, no closing keywords). It runs SAST, secret detection and container scanning, re-runs the claim's rules inside the built image, and replays any pending migration against a shadow database while a writer keeps inserting. If all of that passes, it auto-merges.

On main, a gate decides whether to release, defer or block. The release deploys a canary that gets no traffic but has its own URL, and probes it: in our demo, a partner whose certificate chains to the distrusted root has to be rejected, and it is (HTTP 502, certificate error). Traffic then moves to 50% and then 100% while k6 and a per-revision error-rate check watch it. The release commits the new proof and closes the issue. If any step fails, all traffic goes back to the previous stable revision.

The demo covers two claims. anchor-trusted checks that Node is patched for its line and that no distrusted or weak root is in the effective trust store. migration-safe checks that no blocking DDL is pending against a large table.

How we built it

The engine is Python 3.12 (typer, pydantic v2, pglast for parsing SQL, RFC 8785 JSON for fingerprints, cryptography for weak-key checks) with 196 pytest tests, some of them against a real Postgres 18. The app is Node 24 with Express 5. Its trust policy filters tls.getCACertificates('default') by public-key hash and calls tls.setDefaultCACertificates() before any TLS connection is made. A sidecar plays two TLS partners signed by a demo PKI.

On GitLab we used:

one .gitlab-ci.yml that routes to separate lanes (control loop, MR, main, release);
a dynamic child pipeline for releases;
GitLab's security scanning templates;
auto-merge;
scoped labels as the claim state machine;
a Duo custom flow on the Assign trigger.

On Google Cloud we used:

Cloud Run (a sidecar container, and revision tags for canaries);
Cloud SQL Postgres 18, Artifact Registry and a public bucket for the feeds;
Cloud Monitoring for per-revision errors;
Secret Manager.

CI reaches Google Cloud only through GitLab OIDC tokens and Workload Identity Federation, with no service-account key anywhere.

There's also a public status page on Cloud Run that shows each claim's evidence, its path from issue to MR to release to mint, the live traffic split during a rollout, and what changed in the world.

Challenges we ran into

--use-system-ca adds the OS certificate store to Node's roots; it doesn't remove anything. To actually drop a distrusted root we had to filter the default CA set at runtime.
"Safe migration" depends on table size, which changes without a commit, so we measure it instead of guessing. On 600k rows, a plain CREATE INDEX held a ShareLock for about 1.4 s while writes queued behind it. The CONCURRENTLY version held none.
GitLab rejected our flow prompt because it contained HTML comments, which we use as markers in the issue format. We now describe the markers in words.
Duo flows act under a composite identity: the MR author shows as whoever triggered the flow, while the commits come from the flow's service account. Our merge policy rejected a correct fix because of this until we switched to checking provenance properly.
An MR pipeline runs the MR branch's own code, so a bad MR could weaken its own checks. The jobs that make decisions now load the policy code from the target branch.
On the last day, a fresh end-to-end run found a real race. GitLab started two scheduled control loops 21 seconds apart, which opened duplicate issues, and four release pipelines then overlapped until one deploy replaced another's canary. The rollback did its job, but we fixed the cause: we now serialize the control loop and whole release pipelines. That same run also exposed an auto-merge that stalled after a green pipeline, so the control loop now merges such an MR itself at the same commit.

Accomplishments that we're proud of

A clean run with both claims at once: a Node security release, a root distrust and a table growing past 500k rows, all without a commit; two issues and two Duo merge requests; the first merge was deferred because the other fix was still in flight, and the second merge shipped both; both issues were closed by the release 38 minutes after the world changed.
The trust fix is proven on the canary: the trusted partner still gets 200 and the distrusted partner gets 502, and nobody edited a root file by hand.
Failure handling works: when we forced a canary to fail at 50%, traffic went back to the stable revision, and the next scheduled loop re-released the fix and closed the issue.
The policy holds: a hand-made "fix" that edited the claim files was rejected, the claim was quarantined, and nothing merged.

What we learned

A deterministic program should decide whether something is still true. Writing the fix can be handed to an agent, as long as policy checks and re-proofs stand between the agent and production.

Probes that prove a fix have to hit only the new revision, so we use a tag URL. Checks on the main URL can only tell you about SLOs while most traffic still goes to the old version.

Running the whole thing end to end, more than once, found bugs that unit tests never would.

What's next for Stale

A dependency-trusted claim for the axios kind of incident (lost provenance, a new publisher, new install scripts, very young versions).
A contract-compatible claim for the CrowdStrike kind.
A flow that analyses failed canaries, a scheduled release train, Terraform for the infrastructure, and a root feed built from Mozilla's OneCRL.