Proof of Done

Merged isn't done. After every deploy, a GitLab Duo flow proves each shipped issue in production and files the photo in the issue. 4 of 7 features that passed CI were broken live; it caught all 4.

Comment

After every deploy, a GitLab Duo flow turns each shipped issue's acceptance criteria into browser steps, walks them in production, and files the photograph in the issue: closed with the frame that proves it, or reopened with the frame that breaks it. Of 7 features that passed CI, merged and closed, 4 were broken in production; it caught all 4. Issue #7 was reopened 12 minutes after its merge, fixed by the fixer agent, merged by the guard inside a written policy, and closed 31 minutes after the merge with nobody touching it. On Oct 19, on the current build, a new feature (#10) went from merge to proven fix in 24 minutes the same way. Later, a header that hid the floor plans while the image still answered 200 (an HTTP probe would have passed it) was caught by the nightly light table and rolled back 7 minutes after the deploy (that night's schedule we ran by hand, to see it act). Issue #7 · Proof sheets · Pipelines · The flow in the AI Catalog · Carrel, the demo app · Code on GitLab

Inspiration

An issue closes when its merge request merges. The tests that let it merge ran in CI, against CI's configuration. Nobody looks at production, so on most teams "done" means "merged".

The failures users actually hit live in that gap: a server in another time zone, a feature flag nobody switched on, a missing secret, a security header. Now that agents write more of the code, changes merge faster than anyone verifies them, and the gap grows. It hurts release owners and QA leads most: they sign off on what shipped without seeing it.

We wanted the issue tracker to tell the truth: an issue is done when it works in production, and here is the photo.

What it does

  • Checks the plan before the merge. A plan:criteria job answers each acceptance criterion a change ships, and each planned one, the way the runner will answer it in production: a browser can prove it, it needs a person, or it is too vague to fail. A vague one turns the pipeline amber without blocking it.
  • Proves every deploy. When a pipeline on the default branch passes after a human merge, a GitLab Duo custom flow starts with no human action. Its examiner agent finds the merge requests in the deploy and the issues they close, reads each issue's acceptance criteria, looks at the live pages (their roles, names and labels), and writes a check plan in a fixed vocabulary of 18 step words.
  • Photographs production. A deterministic runner, Playwright in Chromium, walks each plan against the live app and photographs every step. It first waits until production reports the commit it was asked to prove. It also checks what a visitor takes away, for example a downloaded calendar file's start time in the right time zone.
  • Lets only the runner decide. Three outcomes: held (every step happened as written), failed (a step didn't, with the exact sentence of what was seen), needs a person (a browser can't observe it, like an email arriving; never passed silently).
  • Files the evidence in GitLab. The clerk agent closes a held issue with an inline strip of frames, reopens a failed one with the failing frame, and labels the rest for a person.
  • Fixes what it can, under a policy. The fixer agent opens a small fix merge request on a proof/fix-<iid> branch, with a test. A guard job merges it only if it stays within the policy (at most 3 files and 40 lines; no changes to the pipeline, flow, runner or checks; no high or critical SAST or secret findings; a green pipeline). Otherwise it writes why and leaves it for a person.
  • Re-checks the fix in production. After the fix deploys, a CI job replays the exact plan that failed, with no model. If it holds, the issue closes with the new frames.
  • Moves issues only from the record. The agents write notes and fix merge requests; closing, reopening and labelling an issue is done by the board, from the roll that proves it, so no agent can close an issue the record says failed. A regression the light table confirms at night goes to the fixer, the same as a failure right after a deploy.
  • Says who touched it. GitLab shows the board's work under its token's owner, and a merge's own effects under the person who merged. Every issue page on the board has a touch ledger that matches each GitLab event to the merge, the flow's account or a board job, and counts what's left: #7's, from the merge to the close, has no person.
  • Keeps done proven: the light table. Every night it replays every proven check with no model, lays each frame over the last one that held, pixel by pixel, and says how much changed. A failure confirmed on a second replay reopens its issue.
  • Rolls production back, within a policy. If a proven issue regressed and the last deploy is the only thing that changed, a keyless job moves Cloud Run's traffic back to the revision that ran the commit the issue last held on, waits until production reports it, and writes the policy table on the issue. The policy is code: every regressed issue last held on one commit, production changed since, the deploy is at most 7 days old, no other proven deploy in between. It has run for real: on Oct 7 night a deliberate regression of #7 (a Content-Security-Policy header that forgot its own origin) was rolled back 7 minutes after it deployed, and the issue closed again 25 minutes later once the fix forward held (rolls 25 to 27). On Oct 15 it refused, correctly: a check failed on the very commit it had held on, so it said a person was needed.
  • Branches on outcomes, and stops early. The flow's routers end the run at the examiner when there is nothing to prove (the nightly schedule pipeline, a duplicate trigger, a deploy that closes no issue: one model call instead of four), and at the clerk when nothing failed, so the fixer never starts.
  • Keeps a tamper-evident record. Every roll (one run of the checks) goes to the proof-sheet site on Google Cloud Run, hash-chained to the one before it; every note on an issue ends with the roll's hash. Publishing is accepted only from this project's own jobs on the refs they belong on, and from the flow's own account, verified by asking GitLab. Release notes link each deploy to its proof instead of claiming done.

A header hid the floor plans while the file still answered 200. The light table saw it; production rolled back 7 minutes after the deploy.

The demo, for real. Carrel is a study-room booking app we built as the demo product this repository ships. Over three releases on Oct 7, every feature passed its tests, merged and closed its issue, and four of them didn't work in production:

  • Floor plans (#7). The Dockerfile never copied the images, so every plan answered 404. Reopened 12 minutes after the merge; the fixer's one-line fix (COPY assets ./assets) was merged by the guard; the re-check held and the issue closed, 31 minutes after the merge, untouched.
  • Calendar (#2). An 18:00 booking landed at 23:30, because the server runs in UTC while the calendar test pinned Pune time. The fixer used the library's own time function and removed the test's time-zone pin.
  • Share link (#5). The link pointed at localhost:8080, because PUBLIC_URL was set in tests but not in production. Fixed in service.yaml and closed 13 minutes after the failing roll.
  • Waitlist (#3). The feature flag was on in tests but off in production. The flag is fixed; its saved check had been written while the page didn't exist, so the re-check honestly said "needs a person"; a proof-recheck label got it a fresh plan, and it held.

One human click, then the loop: 12 minutes from merge to reopen, 31 from merge to closed, timed on Oct 7.

It also got things wrong, and we kept the record. Two checks failed for reasons in the check, not the app, and the fixer opened plausible wrong fixes; a person closed both, and one would have raised the cloud bill. On Oct 15 a midday replay failed #6 because its saved check expected every room to say "free" and one said "Full today"; the rollback refused, a person challenged the verdict with a label, and the fresh plan held. Each of these led to a safeguard: setup steps that can't count as failures, scoped text checks, clicking links rather than rebuilding them, a plan review before every run, a skeptical fixer that reads the recorded verdict, a proof-recheck label to challenge any verdict, no clock or fullness words in replayed checks, and a guard that refuses cost changes and fixes that would close their own issue. On an app we didn't write (TodoMVC's Dijon example, held to the TodoMVC spec's own sentences), the first two runs filed nothing: the board didn't admit the project, then refused a roll that was too big. Run 1 would also have sent six criteria to a person (the runner had no double-click, hover or focus step), and the clerk reported "nothing failed" when #9 had failed. Run 2 failed #3 wrongly, because TodoMVC hides its "Mark all" checkbox behind a label. Six fixes later, run 3 filed, and its verdicts matched TodoMVC's own known-issues list on 9 of 9 issues (only Routing fails). The fixer then wrote the routing Dijon lacks (2 files, about 25 lines); a person reviewed and merged it, and the re-check held all four Routing criteria in production, so #9 closed with its photos too.

Composite identity, for the record. GitLab shows the fix merge requests as opened by the person whose merge triggered the flow; every fix commit is authored by the flow's service account (Proof of Done), and the in-policy merges (!7, !9, !10) were made by ai-proof-of-done-gitlab-ai-hackathon. !2 and !3 were merged by hand, before the guard existed.

How we built it

  • GitLab Duo Agent Platform: a custom flow (flow registry v1, ambient, public in the AI Catalog, v1.18.0) with three agents, the examiner, clerk and fixer, and conditional routers that branch on fixed answers. Tools: gitlab_api_get (commits, then merge requests, then closes_issues); get_issue, create_issue_note (issue state is moved by the board, from the record); create_commit, create_merge_request; run_command, which calls our runner. A second flow for the Plan stage on new issues, flows/criteria-check.yml, is written on the same tested check, ready to enable with a Work item trigger.
  • Trigger: Pipeline events, Run when Passed, so a human merge leads to the deploy and the flow with nobody in the middle.
  • Agent configuration: .gitlab/duo/agent-config.yml runs flow jobs in our runner image; a setup script prefetches the production deployment's merge requests with the job token; the network policy lists the two exact hosts.
  • The runner (proof-run): Node and Playwright. It validates every plan against the step vocabulary (no custom JavaScript, no leaving the app's origin), photographs each step, outlines pages for the agent, checks calendar files with an RFC 5545 parser, renders the comment strips, publishes and replays runs, and reviews acceptance criteria for the Plan stage.
  • The pipeline: ten CI stages in .gitlab-ci.yml, covering all nine lifecycle stages, all real jobs: plan (plan:criteria, the first stage of every pipeline); verify (three test jobs, 118 tests); secure (SAST and secret detection); package (the runner image to the GitLab registry, the app and site images with Cloud Build to Artifact Registry); release (release notes linked to the proof); deploy (Cloud Run, configured from service.yaml); prove (the re-check after a fix); monitor (the nightly light table and the rollback job after it); configure (configure:drift compares the live Cloud Run service with service.yaml after each deploy and each night); govern (the guard).
  • Google Cloud: Cloud Run hosts the app and the proof-sheet site; Cloud Storage holds the frames; Cloud Build and Artifact Registry build and store images; Workload Identity Federation with GitLab's OIDC ID token lets CI deploy and roll back without a single stored key.
  • The proof-sheet site: Next.js, designed as a photographer's contact sheet: real frames on film strips, a graphite circle on the frame that proves a criterion held, a red cross on the step that failed, a loupe that shows the changed pixels in red. Its API asks GitLab who is calling: a job token must belong to a running job of this project, and a flow token to the flow's own account. Hackathon participants hold the Developer role, so the project can't mint a bot token; the site holds one delegated token in Secret Manager and acts for CI jobs only when its own stored record backs the action.
  • Tests: runner (29), site with the guard and rollback policies, the frame comparison and the footprint model (60), app (29). All run in CI on every pipeline.

Footprint (SCI)

Measured with the Green Software Foundation's SCI (ISO/IEC 21031:2024), per night of keeping done proven.

  • What a roll costs: under 0.1 g CO2e for the browser and the job (measured browser time, Cloud Carbon Footprint coefficients, Google's 2025 region data).
  • What the design saves, measured from GitLab's own records of the tokens each model call used, in every flow session of this project:
    • Model-free replays: the light table replayed 13 proven checks a night from saved plans when this was measured (Oct 15; 11 today); examining them again would take about 63 model calls and 2.2 million input tokens. Saved: 15 to 533 Wh and 6.6 to 230 g CO2e a night, 2.4 to 84 kg a year.
    • The nightly gate (flow v1.14): one model call and 11,001 input tokens on a night with nothing to prove, against 4 calls and 40,358 on each of the 8 nights before. Saved: 0.7 to 7.0 Wh a night.
    • Early stop: a deploy that closes no issue no longer starts the clerk and fixer: 2 calls and about 13k tokens saved each time.
  • The energy of a model call isn't published for this model, so it is a range: low is Google's measured median prompt, 0.24 Wh per call (Elsworth et al., 2025); high is Epoch AI's 2.4 Wh per 10,000-token query (You, 2025), scaled per token. Grid 432 g CO2e per kWh.

Measured from GitLab's own token records: model-free replays save 15 to 533 Wh a night; a night with nothing to prove costs one model call, not four.

Challenges we ran into

  • Developer role only. Participants can't create project access tokens or CI variables. Writes from plain CI (merging a fix, closing after a re-check, reopening a regression) go through the board, which acts only on evidence it stored.
  • GitLab's own closing keywords. The fixer's first commit message, "Fix #3: …", closed the issue on merge, before production was re-checked. The fixer now writes "Repair #3 in production", and the guard refuses any fix that would close its own issue.
  • Scoped labels are exclusive. A proof::recheck challenge was silently dropped when a status label was added, so the challenge label is unscoped (proof-recheck).
  • Flows can't trigger flows, and bots can't trigger flows. So the loop after a fix runs in plain CI: the guard merges the fix and the re-check job replays the saved plan. This turned into a strength: replays need no model, so nightly checks are cheap, repeatable and comparable.
  • A passed schedule pipeline starts the flow too. The trigger can't tell a deploy from the nightly schedule, so the examiner decides from the payload alone and a router ends the flow there.
  • The flow's token can't read deployments. We prefetch the deployment's merge requests in the agent configuration's setup script with the job token.
  • Flow prompts are Jinja templates. Plans use [[name]] for remembered values; {{name}} would be swallowed.
  • Cloud Run reserves /healthz. The version check moved to /health.
  • Making a model's output safe to run against production. A small vocabulary validated in code, an origin lock, and verdicts that only the runner's assertions can give.

Accomplishments that we're proud of

  • A full hands-off loop: merge, deploy, prove, reopen, fix, guard, merge, re-check, close, and a nightly replay that rolled production back on its own. In #7's loop the only human action was the merge; every in-policy fix (!7, !9, !10) merged under the flow's own account; a person challenged a verdict with a label on Oct 7 and Oct 15. The board's touch ledger shows it event by event.
  • Evidence where the team already works. The failing frame is inline in the GitLab issue, and every roll can be opened and verified.
  • A visual regression caught where HTTP said 200.
  • A bug no unit test could find. On Oct 16 the light table failed "Cancel a booking" twice: Cloud Run had run two Carrel instances at once, and a booking made on one was cancelled on the other, which had never seen it. The rollback policy correctly refused (the app hadn't changed). The fix needed a new storage bucket, a person's change, so a person wrote it (!11); after the merge the flow examined #4 again and it held (roll 44).
  • Two fresh runs on the final build (Oct 19). Two new features whose tests passed and which broke only in production, each merged once. #10: the Dockerfile never copied the rules file; reopened 14 minutes after the merge, the fixer added COPY content ./content (!15), the guard merged it, and #10 closed held 24 minutes after the merge, with no person in its touch ledger. #9 found two bugs in our own guard first (it judged a fix on old findings in files the fix didn't touch, and its refusal label replaced proof::fix); after we fixed them, a person restarted the fix's pipeline and the guard merged it. Its ledger lists those human acts.
  • All nine DevSecOps stages, each one a real job or agent you can click in the pipeline history.
  • A tool that says what it can't check. Three outcomes, never two, from the Plan stage to production.
  • It works on code we didn't write. On TodoMVC's Dijon example, with criteria copied from the TodoMVC spec, run 3 closed eight issues with their photos and reopened Routing with four failing frames: 9 of 9 verdicts match the list of known failures TodoMVC's maintainers wrote years earlier (trial project). The fixer then wrote Dijon's missing routing; once a person merged it, #9 held in production and closed (proof sheet).

What we learned

  • The most valuable check is the one CI can't run: production with production's configuration.
  • Agents are good at reading intent from prose and writing structured plans. Deterministic code should judge the results.
  • Replayed checks age: words like "today" or "free" in a saved check fail at another hour with nothing broken.
  • Governance is what makes hands-off acceptable: a written policy, a narrow vocabulary, and a tamper-evident record.

What's next

  • Canary-first proofs: check a Cloud Run canary revision before shifting traffic.
  • More checkable criteria: a reload step, email through a test inbox, API responses, accessibility checks.
  • A Catalog-ready template: one flow, one agent config and one CI include, so any GitLab project gets proven deploys in minutes.
  • A cheaper "nothing to prove": decide it deterministically before the examiner starts, so a docs-only deploy costs no model call.

Built With

  • artifact-registry
  • chromium
  • docker
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • google-cloud
  • google-cloud-build
  • google-cloud-run
  • google-secret-manager
  • next.js
  • node.js
  • oidc
  • playwright
  • react
  • typescript
  • workload-identity-federation
Share this project:

Updates

Submission history