Noxkeep

A night watch for apps whose builders moved on: every night it breaks its own app on purpose, a GitLab Duo flow drafts the fix, a human approves, and only a passing re-drill turns it green.

Comment

Some software keeps running after its builders have moved on: an app an agency handed over, a shop's app built with AI tools over a weekend, a service a team left behind. Nobody checks whether a stranger can read the customer table, whether last night's backup actually restores, or whether a deploy rule quietly lets a schedule push to production. The owner finds out from a customer, or from the news.

Noxkeep is a night watch for that kind of app. Every night, in GitLab CI, it attacks and breaks its own demo app on purpose and measures what a stranger could reach. When a check fails, a GitLab Duo flow decides what to do and drafts the fix, a human approves and releases anything risky, and only a passing re-drill turns the check green. Code measures every number and decides every state; the AI decides, explains and drafts, and a human merges. If Noxkeep itself stops, its certificate goes grey.

One night, start to finish

On 26 October I planted a fault the way a hurried developer would: a table called supplier_terms, added by hand outside the migrations, with row level security off. It held four fake rows, and anyone with the app's public key could read them. Nobody told Noxkeep.

The anonymous-access drill found the table through a read-only inventory login, counted the rows a stranger could read without reading them, and posted a signed, redacted note on the app's care-record issue. The triage job started one GitLab Duo flow session. The flow read the evidence, chose fix_now, and opened a draft merge request adding 0005_enable_rls_extra_tables.sql. Migrations are a risky path, so the fix-scope guard failed it until I posted "noxkeep approve" followed by its exact head commit. I merged it and ran the manual release jobs: release-migrate applied the migration, the demo redeployed, and the re-drill reported readable_tables 1, writable_tables 0, 0 finding(s). The one readable table is the public menu. The certificate went green on the next scheduled run. That loop is the whole project.

What Noxkeep does

Drill. Nine drills feed the certificate, eight nightly and one weekly. Code measures every number:

anonymous-access: rows a stranger can read, and whether a write is accepted; restore (weekly): the newest real Supabase dump restored into an empty PostgreSQL and compared table by table; clock: runtime end-of-life dates and TLS certificate days left; armour: unpinned images, unlocked installs, install scripts left on, committed key files; supply-chain: install scripts and lockfile integrity; supply-chain-probe: installs a package of my own in a throwaway CI job, counts the secret names an install script in that job could see (never their values), and asks GitLab whether the job token could push to the repository. It sees only its own job, not the jobs that build or deploy the app; advisories: OSV and CISA's known-exploited catalog; postmortem: whether a deploy job can run from a schedule; exposed-settings: plain-text secrets in Cloud Run settings, old revisions included.

Decide and fix. In the GitLab Duo custom flow (flows/care.yml), a read-only reader answers fix_now, escalate or wait, and a writer with no merge, approve or start-flow tool opens a draft merge request or a confidential issue. The writer acts before code checks the answer, so the real gate is the merge: fix-scope fails a fix on a risky path until a listed maintainer approves its exact head commit, and a fix migration may only add policies for named roles, row level security, new tables, columns, indexes and named grants, or drop policies and revoke. Anything else is left to a human. After the session, every number the AI wrote must match a measured value by key and value, and a note that breaks a public-text rule is replaced whole.

Prove. After the release, the drill runs again. Only a passing drill turns a check green. The AI cannot.

Certify. A certificate on Cloud Run (a page, JSON, a badge and a results page in plain words) shows one card per drill: green, red, amber or grey. It goes grey on its own if the drills stop: after 36 hours for a nightly drill, 8 days for a weekly one. All nine drills were green after the nightly run at 06:13 UTC on 27 October, the first with the install probe on the certificate.

How it uses GitLab

Duo Agent Platform. Two custom flows sit privately in the AI Catalog with no triggers: care, and Breach Echo, which has an empty toolset and pulls preconditions out of 25 dated public incident records for code to check against measured facts. CI starts their sessions. The flows API refuses the CI job token (HTTP 401), so a protected main pipeline reads a fine-grained token, limited to Duo Workflow create and read, from Secret Manager. Code caps care-flow starts at 3 a day and 1 per drill a day. A third flow, Adopt, is written to onboard another project but has not been uploaded or run.

CI/CD. Pipeline schedules run the drills, and manual release jobs guard the live database. Google Cloud login is keyless OIDC through Workload Identity Federation, with no stored cloud keys; Secret Manager holds the database URLs, the flow token and the note-signing key.

Issues and merge requests. Each drill run posts one signed, redacted note on the care-record issue. Escalations go to confidential issues: the flow asks for one, and code reads it back and makes it confidential if it isn't. Fixes arrive as draft merge requests.

Guards. fix-scope runs on every merge request, and GitLab's "Pipelines must succeed" setting is on, so a failed fix-scope blocks the merge. bot-on-main fails while a flow bot commit sits in the last 200 first-parent commits of main, unless a human reverted it; it detects such a commit, it does not stop the push.

What I checked live on 27 October

The install probe passed on main: 0 counted secrets, and GitLab accepted the read check but refused the push check with HTTP 403, so the job could not push. Only then did it join the certificate.

The Last Key check, which looks for unused cloud keys to retire, read the cloud project's key policy and keys once its account had two read-only roles: 0 user-managed keys, 0 findings.

The response command, which drafts a pin plan when an incident record names a compromised package version, was run against a copy of a real public record naming compromised npm and PyPI versions, with a labelled demo pin that it flags and pins back to the last safe version.

Breach Echo's flow is configured with an empty toolset. The echo-collect job read its session in CI and counted 0 tool calls in 1 model turn, the same count I got reading the session's checkpoint by hand.

What I won't claim

Breach Echo's first scored run does not beat its naive baseline. It got all 7 records it decided right, but it said unknown on 18 of 25, so it scored 7 of 25 against 8 of 25 for a rule that always answers "doesn't apply". 17 of those 18 answers match records labelled unknown, but the scorer never counts an unknown as correct, and I'm not changing the rule to flatter the result. The 25 labels are mine; four were re-judged against the measured inventory before the scored run, and with 25 records the scores are only indicative.

GitLab ignored the read and write privileges I requested for the catalog flow, so what actually limits the care flow is its toolset. The care flow's reader reads job logs and repository files, which may hold text a stranger wrote; its prompt treats that text as data, and no tool enforces it. A merge request that deletes a guard job stops it running, so CI file changes need a human review. Today Noxkeep watches one demo app with faults I planted; pointing it at another project depends on Adopt.

The README marks what I haven't seen live as UNVERIFIED. As of 27 October that includes:

the Adopt flow against a live target; Last Key retiring a real key: the project has no user-managed keys, so the activity check, the independent verifier and decoy retirement have not run on one; what the install probe sees for a project that does allow job-token push (only the refusal has been observed); whether tools were offered to the Breach Echo model at all.

Try it

The repository and the live certificate are linked under "Try it out". In the repository, the README's "Judge, start here" section is the five-minute path, and "Run it on your project" lists what you need to set up.

Why the grey matters

Built for GitLab Transcend, "Life After Code", 2026, under the MIT licence, and deployed on Google Cloud Run. If Noxkeep itself stops running, its certificate turns grey within 36 hours. Silence never passes for health.

Built With

  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-cloud-run
  • google-secret-manager
  • node.js
  • postgresql
  • python
  • react
  • supabase
  • vite
  • workload-identity-federation
Share this project:

Updates

Submission history