Second Key

A release gate for Uptime Kuma: CI gathers evidence, code signs the verdict, GitLab Duo agents explain it and propose fixes, Cloud Run ships in steps, and a person turns one key per commit.

Comment

Inspiration

Uptime Kuma is a popular self-hosted monitor with a small maintainer team and a long queue of contributor pull requests. Writing a change is no longer the slow part; deciding whether a change is safe to ship is. Its maintainers even added an AGENTS.md asking coding agents to stop sending low-effort changes. We wanted the opposite kind of agent help: one that never writes the contributor's code, but does the checking a maintainer would do and leaves one decision to a person.

What it does

When a merge request opens, GitLab CI builds an import graph and runs only the tests that can reach the change, falling back to the full suite whenever reachability is unknown. It mutates the touched lines, fuzzes three Uptime Kuma functions, runs GitLab SAST, Secret Detection and Dependency Scanning on the changed lines, applies diff rules for removed auth guards, removed escaping, shell execution and weakened tests, and looks up earlier reverted or fixed changes on the same files. Code, not a model, turns that evidence into GO, NEEDS_REVIEW or BLOCK, shows blockers at file:line in the Code Quality widget, and signs an in-toto statement for the exact head commit with cosign keyless.

The pipeline event starts the sk-verdict-note GitLab Duo flow. It explains the verdict in an Evidence Note: the one thing to look at first, every reason with its evidence, the code lines no test noticed changing, and what would change the verdict. On a BLOCK it files an issue per blocker and, where the fix is local, opens a draft fix MR. On !87 the gate blocked a removed checkLogin that every test missed; the flow opened !88 restoring exactly that line, and the same gate gave !88 GO. The agent gets no special treatment: its fix is held to the blocked MR's files by code, judged on the combined change, and needs its own key.

A person turns the second key for one head commit, only after that commit's evidence is signed. On the merge, verify-key checks the signature, that the landed diff is the one that was evaluated, and the key. Then the signed image goes to Cloud Run as a revision with no traffic and moves to 5%, 25% and 100%, each step held only while Cloud Monitoring shows it inside its SLO; a breach puts all traffic back on the revision recorded before the rollout. When a scheduled pipeline on main failed, the sk-incident flow started on its own, opened incident #4 and wrote a root cause that kept the facts from the logs apart from its guesses.

The gate also checks itself and its agents. A change to its own policy is replayed on every recorded eval MR and blocked if it loses a catch. Every note a flow writes is re-read against what CI recorded. Five trust invariants run on every pipeline (the key comes only from a person, the verdict is a pure function of evidence and policy, each agent holds only the write tools its step needs, labels grant nothing, the judging code is the default branch's copy). A claims gate fails CI if a sentence in our docs disagrees with the proof file it cites.

How we built it

Everything is in the GitLab repository: .gitlab-ci.yml (evidence jobs, gate, attestation, verify-key, release, rollout), secondkey/ (a Python package with 221 unit tests: test selection, impact graph, diff rules, history signals, verdict engine, attestation, verify-key, fix proof, trust audit, Cloud Run rollout and SLO gate), policy/ (the scoring policy, key holders and agent capabilities), .gitlab/duo/ (four custom flows published in the AI Catalog, plus review instructions for GitLab Duo Code Review Flow) and proof/ (every published number, regenerated byte for byte by make proof). The evidence tools and the policy always run from the default branch's copy, so a merge request cannot change the code that judges it.

On Google Cloud, CI signs in with Workload Identity Federation (only this project's protected branch, no key file), Cloud Run pulls the signed image by digest through an Artifact Registry remote repository, the rollout job runs cosign verify before deploying, slo.py reads Cloud Monitoring per revision, and the admin password lives in Secret Manager. Every job that moves production traffic shares one GitLab resource group, and a cancelled step restores the revision it recorded.

Challenges we ran into

For most of the build we thought custom flows needed the Maintainer role; the organizers pointed out the hackathon role can create them once the hackathon group is the default Duo group. CI has no token that can write notes, so the second key was designed around what a pipeline can read. The first flow sessions failed in instructive ways: the notes tool strips HTML comments, so our hidden verdict marker never reached the model; the explainer invented thresholds the marker did not carry. Each fix became a versioned catalog item and a decision record, and we built an audit that re-reads every flow note against CI. The Google Cloud project was suspended for review for most of the hackathon; when it came back we learned that a flow does not re-trigger on pipelines its own account caused, and that Cloud Monitoring lags about two minutes, so each SLO step drives traffic and waits before it counts.

Accomplishments that we're proud of

A pre-registered evaluation on real bugs: 30 bugs Uptime Kuma had already fixed, each fix undone on main by a rule sealed (and tagged) before the first MR. "Merge when the tests pass" stopped 1 of 30, and 4 with the whole suite re-run. Second Key flagged 17, and 16 of those had passed every test. The cost is honest too: 35% of good changes go to a reviewer, and a random reviewer at that rate flags 17 or more with probability 0.012.

The agent loop runs end to end on real MRs (!87 blocked, !88 fixed by the flow, judged GO), the fuzz job found a real crash in Uptime Kuma 2.5.7 that we fixed in !33, and the first CI rollout held 5% on 33 requests, 25% on 243 and 100% on 1107 with no 5xx, while the rollback drill caught a revision answering 500 in its first minute.

What we learned

Mutation testing on the touched lines carries most of the signal: without it, 8 of the 30 bugs are flagged instead of 17. We also published what a careful judge would find anyway: of the 13 bugs that got GO, 8 change only Vue components, where mutation testing has nothing to mutate, and on this set a single number (blast radius ≥ 2, picked after seeing the data) flags 19 of 30 at a lower review load. And agent output needs checking like any other input: a model can ignore a rule in its prompt but not a rule in code, which is why the fix proposer's scope is now enforced by code.

What's next

A paired forward experiment with real good and bad changes on the same code areas, per-MR preview revisions with replayed traffic, a shared database so canary revisions share state, a second repository in the evaluation, and a required pre-merge status check where the role allows it.

Built With

Share this project:

Updates

Submission history