HoldFast

Holdfast stops hijacked dependencies in GitLab: a CI gate blocks the release, Duo agents explain and fix the merge request, and a human makes the final call.

Comment

Holdfast

Last trusted publisher. Held.

Inspiration

October 22, 2021. Someone takes over the npm account behind ua-parser-js, downloaded more than 7 million times a week, and publishes three versions from the real maintainer's account. Each runs a preinstall script that installs a cryptominer, and on Windows a password stealer. They were live for about four hours.

It keeps happening. In 2018, event-stream was handed to a stranger who hid a payload aimed at a Bitcoin wallet. In 2025, the Shai-Hulud worm compromised more than 500 npm packages in September and about 800 more in November. Sonatype logged 512,847 malicious packages in a single year.

None of these began with a CVE. Each began with someone new controlling a package people already trusted. A vulnerability scanner asks "is there a known bug?" and finds nothing in the hours that matter.

I discussed the idea with engineers in my circle of friends and colleagues, and the same gap kept coming up: teams have scanners and dependency bots, but nothing that stops a release because a package they already trusted quietly changed. Holdfast fills that gap with a simpler question on every merge request: is this still the package we reviewed?

What it does

Holdfast keeps a ledger in Git of what your team reviewed for each dependency: publisher, owners, install scripts and checksum. Every pipeline checks the lockfiles against it, across npm, PyPI, Maven, Go, Cargo, RubyGems, NuGet and Composer. If something changed that nobody reviewed, the merge is blocked and the release is held. GitLab Duo agents explain and fix it, and a person makes the final call.

When GitLab Duo flow
An MR opens MR Guardian quotes the custody report for the exact commit
Custody fails (exit 3, plain code, no AI) Remediation Clamp commits an exact pin back to the reviewed version, touching only the bad package
Custody fails Incident Commander runs an investigator, risk analyst and auditor, and posts one evidence-linked docket
Someone wants to ship Release Readiness gathers the evidence and stops at a human checkpoint
A breach issue is filed Incident Docket drafts it, a second agent checks every claim, and a human approves
You want new attack tests Adversarial Drill opens a Draft MR of regression fixtures

A release that truly needs a new version gets an Unseal: an exact fingerprint, a named approver from an allowlist, at most 30 days. Bots and the change author are rejected.

What the check detects

Signal Severity Real attack
lifecycle_hook_added: new install-time code HIGH ua-parser-js, Shai-Hulud
unreviewed_version: a version nobody reviewed HIGH Any surprise bump
publisher_change, owner_change HIGH A package handed to a stranger
checksum_drift: same version, different bytes HIGH A republished artifact
owner_added: new publish rights MEDIUM event-stream

In the ua-parser-js attack no owner changed, so an owner check alone would miss it. Holdfast catches it because 0.7.29 adds a preinstall script the reviewed 0.7.28 never had. Holdfast doesn't judge intent; it holds whatever nobody reviewed, and a person clears legitimate upgrades through a CODEOWNER-approved ledger change.

The demo: MR !29

A bot guard for a Node gateway, a safe dotenv update, and ua-parser-js 0.7.29.

  • Tests pass. Custody fails with lifecycle_hook_added and unreviewed_version. The merge is blocked. The sweep only reads the lockfile, so the malicious script never runs.
  • MR Guardian reviews it on its own: "It is not a merge approval, release certification or Unseal."
  • Remediation Clamp puts only ua-parser-js back to 0.7.28 in package.json and the lockfile: "No ledger, policy, CI or source changes." The feature and the dotenv update stay.
  • Incident Commander files a docket marked CONTAINED_PENDING_HUMAN_REVIEW: "Holdfast's own CI decision remains authoritative."
  • The rerun goes green. Release Readiness audits the MR and stops at a human checkpoint.
  • The reviewer merges. The feature reached main. The hijacked package never did.

One MR per ecosystem

The same flow ran on one more MR for each of the other seven ecosystems, each with a real feature, a safe update and one unreviewed bump. Every one was held, got a single Clamp commit that reverted only the bad package, and went green with the feature intact. They're all linked from the Pages site.

How I built it

  • Deterministic engine in Python with no runtime dependencies. One "grip" per ecosystem parses the lockfile, works out who controls each package, compares it with the ledger and writes an exact pin back. Each finding's fingerprint is tied to package, version and bytes.
  • Six GitLab Duo flows on the Agent Platform, each with its own service account, started by MR, pipeline, mention and issue-template events.
  • An MCP server with eleven read-only or preview-only tools. A patch that touches the ledger, policy, CI or enforcement code is rejected.
  • GitLab CI/CD with per-ecosystem tests, MCP tests, attack replays, SAST and secret detection, then the sweep, quarantine, certification and release gates.
  • Attack replays: event-stream, ua-parser-js and Shai-Hulud run on every pipeline. All three are caught.
  • Watchdeck on GitLab Pages shows the live release decision, findings, replays, the live MRs and an interactive demo.

Challenges I ran into

  • The attack that changes no owner. A takeover sounds like a change of owner, but ua-parser-js was published from the real maintainer's account. Comparing publishers and owners alone would have let it through. The ledger had to cover everything a release can change: install scripts, bytes and the reviewed version. An unreviewed version first counted only as a warning, which meant a version bump could slip past, so it now holds the release by default.
  • Eight ideas of "owner". npm has publishers, PyPI hides the uploading account, Maven and Go identity is a namespace, and a Go replace can silently point a module at someone else's fork. Each grip had to define "same custodian" for its registry and state its blind spots honestly.
  • An agent that can fix, but can never approve itself. A comment-only agent is safe and useless; an agent with write access can quietly "fix" a failure by editing the ledger or the CI file. I drew a hard boundary in code: agents get read-only and preview-only tools, any patch touching the ledger, policy, CI or enforcement code is rejected, and holdfast duo validate-flows refuses to load a flow that writes without a human checkpoint.
  • A fix that looked done but wasn't. Clamp's first npm fix pinned package.json but left package-lock.json on the hijacked release, so npm ci would still have installed it and the next sweep still failed. The fix now restores the lockfile entry and its integrity hash too, and the pipeline must pass again before anyone reviews it.
  • Registry data is attacker-controlled text. A malicious package description could carry GitLab quick actions or hidden instructions into an agent's note. Every value from a registry is now treated as untrusted data, with newlines, backticks and leading slashes neutralized before it reaches a comment.
  • My own reviewer broke my gate. When MR Guardian reviewed Holdfast's 0.2.0 release, it found six real issues, including a way for a Duo service account to approve an Unseal and an Unseal that silently covered later releases. All six were fixed in 0.3.0 before it shipped.

Accomplishments that I'm proud of

  • It caught the attack nobody's owner check would. ua-parser-js 0.7.29 came from the real maintainer's account, and Holdfast still blocked it on a real GitLab merge request.
  • The full loop runs on its own until the final call. In MR !29 a failed pipeline woke three Duo flows, Clamp committed a minimal fix, the rerun went green, and a person merged. The feature shipped; the hijacked package never did.
  • Proven in all eight ecosystems, not just npm. Every live MR was held and fixed by a single commit that reverted only the bad dependency, with the feature code untouched.
  • AI never decides what ships. The release gate is plain code over files in Git, and it fails closed: if Duo is down, the release stays held.
  • The agents audited their own maker. Duo found real security bugs in Holdfast's gate before release, which is exactly the job I built it to do for other teams.
  • Drops into any GitLab project as a CI component, with zero runtime dependencies and no paid service.

What I learned

AI is great at the slow parts: reading logs, lining up evidence, writing a minimal patch and explaining it. It shouldn't decide what ships. The design that held up: the agent proposes, the pipeline checks again, and a person signs off.

Questions reviewers ask

  • Can someone just edit the ledger? It's CODEOWNER-protected, agents can't write it, and every certificate records its SHA-256.
  • Won't this block every upgrade? Only unreviewed changes to watched packages, on purpose. A reviewed upgrade is recorded in Git with a named human.
  • Can the install script run first? Not in the check: it installs nothing. Projects that install in CI should wait for the gate or disable scripts until custody passes.
  • Live or replayed? MR pipelines replay recorded registry evidence so results are deterministic. A scheduled job checks the live registries.

What's next for Holdfast

  • Sigstore and SLSA provenance as ledger evidence.
  • Scheduled live-registry sweeps, to catch a hijack between merges. Four hours was enough for ua-parser-js.
  • A shared trusted ledger across GitLab groups and instances.

Demo

Sources

Built With

Share this project:

Updates

Submission history