Leakfix

When a password leaks, GitLab kills it, replaces it, and proves it.

Comment

Inspiration

Secret scanners are good at telling you a password leaked. They do not make it stop working. Until someone rotates it by hand, anyone who saw the push can log in. Leakfix closes that gap inside GitLab, with one human decision kept where it belongs: the merge.

What it does

When someone pushes a database password to main, most tools tell you about it. Leakfix ends it.

In the same pipeline where the secret gate fails, a CI job signs in to Infisical with GitLab's own OIDC token (no stored key), changes the Postgres password, saves the new one in Infisical and checks two things: the new password works, and the leaked one is refused. In one recorded run that took 45.9 seconds from the leak commit, most of it CI job start-up.

Then GitLab Duo takes over the paperwork. Flow 1 opens an incident and a fix MR that removes the file and records the incident. A human reviews and merges: that is the one decision we keep human. The merge runs a verify job that logs in with the production credential from Infisical and tries the leaked string from git history one more time. It is refused. Flow 2 posts the full audit timeline on the incident, with a link for every row, and closes it.

Recorded run on 27 Oct 2026:

  • Leak pipeline 2992566477: gate failed at +12.5 s, leaked password refused at +45.9 s.
  • Incident #3 opened at +120 s, MR !2 opened at +137 s by Flow 1.
  • Human merged at 06:02:51 WAT (05:02:51 UTC). Verify pipeline 2992795648 passed; the leaked string was refused again at 06:03:39 WAT (05:03:39 UTC).
  • Flow 2 posted the audit note and closed incident #3 at 06:04:37 WAT (05:04:37 UTC).
  • Infisical secret version 4 (leaked) to 5. AUDIT result PASS.

How we built it

  • GitLab CI runs secret-gate (gitleaks 8.30.1 with a custom rule for Postgres URLs) and, on main only, rotate-credential with when: on_failure, so containment never waits for an agent or a human.
  • Keyless secrets: the job's id_tokens ID token signs in to Infisical Cloud EU through OIDC Auth. No credential sits in CI/CD variables. A wrong-audience token is refused with HTTP 401 (jobs 17483889114 and 17483905022).
  • Rotation (scripts/keyless.py, standard library plus psycopg): ALTER ROLE with a 40 character password, end live sessions, write and read back the new version, log in with it, and confirm the old password fails with "password authentication failed". It writes a receipt artifact.
  • GitLab Duo flows: Flow 1 (trigger: pipeline failed) opens the incident, branch, commit and MR. It never sees a credential. Flow 2 (trigger: pipeline passed, after the merge) turns the AUDIT line into the incident timeline.
  • Proof (scripts/verify.py, scripts/audit.py): builds the timeline from GitLab timestamps and the rotation receipt, with an evidence link per row.
  • Unit tests run in CI with fakes for Infisical and Postgres, no network and no credentials.
  • Two rotation smoke runs before the real leak (pipeline 2991741674, 26 Oct): leaked password refused 4.244 s and 9.165 s after rotation start, 0 sessions ended, keyless login 635 ms and 621 ms.

Challenges we ran into

  • Our own gate missed a leak. The first real leak used a 27 character lowercase-and-digits password. Its entropy (2.74) was under our rule's 3.0 floor, so the gate passed. We removed the floor and the minimum length, and allowlisted obvious placeholders (password, changeme, ${VAR}, <...>, xxx) instead.
  • Flows cannot start flows, and bot actions do not fire triggers. The human merge became the bridge between Flow 1 and Flow 2, which is also the right place for a human.
  • Database permissions: a role created in the Neon console could not be changed by the owner role. We recreated it in SQL and added a guard that refuses to rotate if the owner URL would log in as the role being rotated.
  • Where the secrets lived: the first keyless runs looked in the wrong Infisical environment. A diagnostic that lists secret names (never values) found it in one run.
  • Honest timing: our first audit note dated detection at the pipeline's finish (+49 s) instead of the gate job's own failure (+12.5 s). We fixed the timeline builder and kept the note as it was posted.

Accomplishments that we're proud of

  • A real end-to-end run on 27 Oct 2026: a live database password pushed to main stopped working 45.9 s later, before anyone looked at it.
  • No credential stored in GitLab. CI reaches the secret store with its own OIDC token.
  • Every step of the audit timeline links to the GitLab job, pipeline, incident or MR that proves it. Evidence: docs/EVIDENCE.md.

What we learned

Detection is not the hard part. The hard part is proving the old credential is dead, in a way a reviewer can click through. Putting containment in CI and the paperwork in Duo flows kept each part simple.

What's next

  • Cut the 27 s between job start and rotation. The biggest piece is pulling the python:3.12 image (11.7 s in job 17490370806); a small prebuilt image with git and psycopg would remove most of that.
  • More targets than Postgres (not implemented yet): cloud keys and API tokens that support rotation.
  • Rewrite history on the leak branch automatically, as a separate MR the human approves.
  • Rotate leaks pushed to branches other than main. Today the gate fails there and Flow 1 opens an incident, but the Infisical identity accepts tokens from main only, so nothing is rotated.

Built With

Share this project:

Updates

Submission history