Inspiration

Deleting a leaked API key feels like the end of an incident. My first experiment said otherwise: a Gemini key I deleted was refused once, then accepted again 25 seconds later. In a later drill the CI runner saw Google refuse a deleted key twelve times in a row, while Europe was still answering it with real data 111 seconds after the deletion. Deleting a key is a request, not a result, and nobody checks the result. I wanted a responder that does not stop at "deleted" and treats Google's own answer as the only proof.

What it does

When a Google API key leaks into a commit on GitLab, VibeGuardian answers inside the pipeline. GitLab secret detection flags the key, in the classic AIza format or the newer AQ. format that Google AI Studio issues. A policy file, read from the default branch so the leaking push cannot loosen it, decides whether the retirement may run on its own or must wait for someone to approve the leak-approve job. The responder finds where the key is used (Secret Manager, Cloud Run), makes a replacement with the same restrictions, rolls every service onto it, checks each one serves the new key's fingerprint, and only then deletes the leaked key. If any step fails it rolls everything back and leaves the old key alone.

A separate job, with an identity that can do nothing except call five probes, then asks Google from the CI runner and from Cloud Run services in Belgium, Tokyo, Sydney and São Paulo until every place has refused the key twelve times in a row. A third job reads Google's per-key request counts to say who used the key while it was public, by region. The incident goes to GitLab Observability as a trace. The gate keeps the pipeline red while the dead key is still in the code, and that red pipeline starts a GitLab Duo flow: a commander agent writes the incident issue from the job records, with fingerprints only, and a fixer agent opens the merge request that removes the key. A person merges it.

The run in my video is drill 15, recorded on 21 October 2026: the leaked key was deleted 77.9 seconds after the push and proven dead from five places at 193.9 seconds (60 of 60 refusals, no flaps); the demo app served 542 of 542 requests OK and was on the new key 5 seconds before the old one was deleted; the Duo flow opened the incident at 297.6 seconds and the fix merge request at 341.4 seconds. Every number I quote has its raw log in the repository.

Before a leak, a pre-commit hook stops a commit that adds a Google key (and OpenAI, Anthropic, Hugging Face and Perplexity keys, with where to revoke each), and a small Windows tray app tells you whether a key on your clipboard is live and can retire it.

How I built it

VibeGuardian is a Python package and command line (vg) driven by a GitLab CI pipeline. No Google key is stored in GitLab: every job trades its GitLab ID token for a short-lived Google token through Workload Identity Federation, and each job gets its own narrow identity. vg-retirer can create and delete keys and roll Cloud Run but cannot reach the probes, so it never grades its own work; vg-prover can only call the probes; vg-watcher can only read Cloud Monitoring. The five probes are a small service built in CI, scanned by GitLab container scanning (0 findings on a Chainguard base) and deployed to four regions as a GitLab environment. The GitLab Duo flow (commander and fixer) runs on the Duo Agent Platform, triggered by the failed pipeline; before the agent starts, any key in its checkout is replaced with a placeholder and the flow refuses to start if one survives. Incidents are exported to GitLab Observability over OTLP.

It covers all nine stages: Plan (incident issues), Create (the pre-commit hook and the fix merge request), Verify (unit tests, an identity check per CI identity, the gate), Package (the probe image built in CI), Secure (secret detection, container scanning, redaction before the agent), Release (the probe service deployed to four regions), Configure (keyless federation, one identity per job), Monitor (who used the key, and traces in GitLab Observability) and Govern (the policy file, the approval gate, the approver on the record).

Challenges I ran into

Fifteen live drills found real bugs, and each is recorded next to its fix: re-runs left orphan replacement keys, a template change stopped the pipeline from being created at all, my own probes were first counted as outside use, refused calls were first not counted, two pipelines raced on the same key, and AQ. keys needed a key-binding permission the retirer did not have. Google's per-key counts can also undercount: in drill 15 the stand-in outsider made 5 calls and Google's complete counts show 3, so I treat them as a lower bound. I also measured that a stolen service-account token kept working for 59.7 minutes after its key was disabled, while disabling the account stopped it in 11.7 seconds.

Accomplishments that I'm proud of

Zero downtime on every retirement I measured, and a proof that would have caught a deleted key still answering in Europe after a single-place check had already called it dead.

What I learned

A deleted key is not a dead key until every region says so, the code that retires a key should never be the code that proves it, and an agent can write the incident and the fix without ever seeing a key value.

What's next

A GitHub Actions version, the leaked service-account key path in the pipeline (it works from the command line today), and the tray app shipped with its engine.

Built With

  • chainguard
  • cloud-monitoring
  • docker
  • electron
  • gemini-api
  • gitlab-ci
  • gitlab-container-scanning
  • gitlab-duo-agent-platform
  • gitlab-observability
  • gitlab-secret-detection
  • google-api-keys
  • google-cloud-run
  • google-secret-manager
  • opentelemetry
  • python
  • workload-identity-federation
Share this project:

Updates

Submission history