Spec-to-Prod Closer

GitLab Duo builds from a spec issue; CI proves every acceptance criterion on the live site before promoting to production, and posts a job-linked verdict back on the issue.

Comment

Try it in three minutes (no login needed)

The metric, in one sentence. A Closed Run is a spec issue taken from issue to production in which every acceptance criterion was verified on the live site, with the merge approval as the only human act after the issue was assigned to the agent, plus at most one @mention retry if a live check failed. Baseline: in our first real Duo run (issue #3) the agent stopped at a draft merge request with a passing check job, 0 criteria verified live. In our seven Closed Runs, every criterion was verified live on staging, and in the latest ones also on production after an automatic promotion. A person acted 2 times in six Runs (assign, approve) and 3 times in Run #9 (plus one @mention retry): 15 in all. Runs counted: nine spec Runs (#4 to #12): seven Closed Runs (Run #12 included: its API dump is recorded and replays as CLOSED, and the CI job 17399378073 judged it CLOSED), one refused by design (#6), one abandoned unmerged (#8). Verdict rule, in short: a Run counts as Closed when the recorded API dump replays as CLOSED. Run #11 is the one exception to read carefully: its first CI verdict crashed (401), we fixed the Closer and re-ran it, and we count the re-run job 17362765396 (pipeline 2973785777, commit adb87a01, naming merge commit 3460dc6f). The Duo flow applies a stricter rule (SPEC S19: the verdict job must run in the pipeline of the merge commit itself), so it would not count #11. Every other Closed Run passes both rules.

Not a CD smoke check. A usual CD setup (GitLab environments with a deployment approval, plus one generic smoke test) decides who may deploy and checks one fixed thing afterwards; here every acceptance criterion of the spec issue gets its own live-check job on the running site, production is promoted only if all of them pass, and the CI job close:summary judges the Run with a job link for every claim or Unknown. Where the live-check:AC-n jobs come from, stated exactly: in the seven Closed Runs they were not generated from the issue. The repository owner wrote the expected values of each Run into scripts/live-check.sh before the issue was assigned (one owner commit per Run, listed under "Human effort per Run"), and the Closer only matched the issue's criterion ids to those job names; a criterion without a job is Unknown. Since 2026-10-25 a CI job plan:live-checks generates the jobs from the Then GET <path> body equals/contains/status lines of the issue as child pipeline jobs. Measured once, on a probe issue (#14, no agent, not one of the Runs): pipeline 2985034631 ran plan:live-checks (job 17440715738), then live-check:AC-1 to AC-3 (jobs 17440722768 to 17440722770) and prod-live-check:AC-1 to AC-3 (jobs 17440727546 to 17440727548). A criterion written as free prose is Unknown, not a guessed check. Run #15 (2026-10-25) is the first Duo Run that used the generator, with no owner commit to scripts/live-check.sh: see "Run #15" below.

How the Closer flow starts. Two ways, both real. An @mention comment on a finished issue started session 9927854 (issue #12). Since 2026-10-24 a Duo trigger on the event Pipeline events: Passed also starts it with no @mention and no person: session 9989569 (CI job 17429361602) started by itself when pipeline 2983029614 went green on main. That first session ended without a verdict, because a pipeline payload names no issue ("No issue was named, so I ran nothing and posted no note"). SPEC S20 and G15 closed that gap with a lookup (pipeline commit, merged merge request, the issue it closes), tests first (7 red, then green). Session 9989781 (CI job 17429470958, after pipeline 2983044817) found that this pipeline closed no issue, answered Run: none and posted nothing. A third push-triggered session, 9989913 (CI job 17429533336), answered the same Run: none. On live data the same command resolves pipeline 2978743135 to issue #12 and prints Run: CLOSED (run from a shell, not a flow session). Seen on Run #13 (2026-10-25): after the merge of MR !14 (merge commit e99bba2b) pipeline 2984929006 passed on main and the trigger started session 10006302 (CI job 17440120322) with no @mention and no comment: it found issue #13 from the pipeline and posted note 4082997322 on it, quoting CI job 17440105819 (close:summary, success): Run: CLOSED. The session page lists @xorjf1027, the pipeline's user (the account that merged), as the one who triggered it; nobody wrote a mention. Run #13 is not counted in the 7 Closed Runs (its API dump is not recorded): https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/work_items/13 and https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/pipelines/2984929006 . Run #15 (2026-10-25), the first Duo Run with generated live checks (issue #15 https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/work_items/15, merge request !16). The issue was created with Duo Developer as assignee (human act 1) and its criteria were GET /run15.txt body equals "run-15", status 200, and GET /build.txt body contains "build". Duo session 10015859 pushed the failing check first (commit 2296615e, merge request pipeline 2986371675, job test 17447552300 failed on purpose), then the files (commit 55af4b61, pipeline 2986382351, test 17447617127 passed). A person approved the merge once (human act 2, merge commit 5c429c2a, 2026-10-25 11:22Z). On main, pipeline 2986389394 succeeded: plan:live-checks 17447655124 generated the jobs from the issue's Then lines, with no owner commit to scripts/live-check.sh; staging live-check:AC-1 to AC-3 were jobs 17447672915 to 17447672917, promote:production was 17447655127 (no manual job), and prod-live-check:AC-1 to AC-3 were jobs 17447678176 to 17447678178 (PASS AC-1 GET https://showcase-f30a54.gitlab.io/production/run15.txt status 200, body 'run-15'). The CI job close:summary 17447655129 wrote Run: CLOSED ("All 3 criteria verified live, 1 human merge approval (safety gate), 0 other interventions"), unit tests 331 passed (job 17447655122). The Pipeline events: Passed trigger started three Closer sessions (10016014, 10016019, 10016027, all Finished); one posted a note on the issue, 4085107534, quoting the CI verdict; the other two posted nothing. The issue carries 13 notes in all, 9 of them system lines and 4 written by agents (including the Developer flow's known error note "Unsupported work item action for Developer goal template: created", which does not stop the session that the assignment starts). Run #15 is not counted in the 7 Closed Runs (its API dump is not recorded in the repository): the count of 7 and the 15 human acts stay as they were. GitLab Credits showed 65.83 credits used in the billing period of Oct 1 to 25 (the page shows no remaining balance).

The Developer flow is still started by assigning the issue (our "work item created" trigger fired but started no session).

Inspiration

Code is cheap. Everything after it — tests, review, deploy, live verification — is not. Small teams and solo builders who ship on GitLab. A verified production, not just a green pipeline. GitLab's own keynote puts it at about 52 minutes a day that a developer spends writing code. We are a solo builder, and our own problem was the last step: an agent says "done" when the pipeline is green, and nobody has checked the deployed site. We wanted the agent's "done" to mean "verified in production", and to say "Unknown" when it was not.

How it compares with what exists

Existing What it does What it does not do What only Spec-to-Prod Closer does
Usual CD setup: GitLab environments with a deployment approval and one generic smoke test decides who may deploy; checks one fixed thing after the deploy does not derive a check from each acceptance criterion of the spec issue one live-check job per criterion, promotion only if all pass, and a CI verdict with a job link per claim or Unknown
GitLab Developer flow (docs) converts an issue into a merge request stops at the merge request; in our first run it told us "It checks the file text, not a real browser render" pipeline-level live checks on staging after the merge, per acceptance criterion
GitLab Fix CI/CD Pipeline flow (same docs) diagnoses and repairs failed jobs does not check the deployed site a failed live check blocks promotion and is reported as a failure, with one agent retry
GitLab Code Review flow (same docs) reviews the merge request works before the merge verification after the merge, then automatic promotion to production only when staging passed
Compliance Sentinel, previous edition (winners) scores each merge request for compliance risk and blocks the merge a gate before the merge a gate after the merge, on the running site
Time-Traveler, previous edition (same page) runs a migration against a safe copy of production tests a change on a copy promotes the real change automatically, but only after live checks on staging, then checks production again
LORE, previous edition grand prize (same page) eight agents that capture project history knowledge capture, not delivery delivery loop closed with evidence

Our one line: the other tools stop at "the pipeline is green"; ours stops at "the live site passed, and here are the jobs that prove it". (Based on the docs and winner write-ups we read, not an exhaustive survey.)

What it does

  1. A person writes one spec issue with Given/When/Then acceptance criteria and assigns it to the Duo Developer agent.
  2. The agent pushes a failing check first. The merge request pipeline goes red on purpose. Then it adds the files and the pipeline goes green.
  3. A person approves the merge. This is the only human act after assignment.
  4. The pipeline deploys to staging and runs one live check per criterion against the running site, not against the tests. In the seven Closed Runs the live-check:AC-n jobs were written by the repository owner, not generated (the expected values of each Run go into scripts/live-check.sh in one owner commit before the issue is assigned: 1564f8b8 for Run #7, 41134ee1 for #9, 91994303 for #10, 12418682 for #11, 74566ba0 for #12), and a criterion without a job is Unknown; Run #15 used the generator instead, see below). That owner work happened before each issue existed and is not in the 15 human acts.
  5. If every staging check passes, the pipeline promotes to production by itself (no manual job) and checks production live again. If any check fails, promotion is skipped.
  6. The last pipeline stage runs the Closer as the CI job close:summary (added after our first five recorded Runs). It judges the Run and posts a summary on the issue when a project token is set; without a token it still runs, reads the public API and leaves the summary in the job log and an artifact. On Run #11 it ran on GitLab with no token and judged the Run CLOSED (see Challenges). Every claim links to a job, or the summary says Unknown.

Flow diagram: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/blob/main/README.md

Failure and recovery are part of the workflow, not an appendix:

How we built it

  • GitLab Duo Agent Platform. The Duo Developer agent (developer/v1 flow) does the coding. 19 Duo sessions on this project, all finished (13 Developer flow, 6 Closer flow; counted on the Automate > Agent sessions page on 2026-10-24, before Run #13, login needed to open). Without login, the work is visible as commits authored by "Duo Developer" on merge requests !1 to !13, for example https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/merge_requests/13/commits (commits 4a5f22da, the failing check first, and fdc1fead, the files). The author is the Duo Developer service account; the committer field shows the account that started the session.
  • What Duo does in a Run, counted from the recorded Runs (python -m infrastructure.measure_cli --duo, output copied into the README): across the 7 Closed Runs, 15 of 15 commits in the merge requests were authored by Duo Developer and none by a person, Duo pushed 1 fix commit on main (Run #9), and the merge requests changed 27 files in all. The Closer is also a Duo custom flow, flows/spec-closer.yml (flow id 1019281), with three tools and no file-editing tool: it quotes the verdict of the CI job close:summary and never judges by itself (real receipt: issue #12, session 9927854, note 4072210044, Run: CLOSED). The Developer flow is started by assigning the issue (our custom trigger "work item created" fired but did not start a session); the Closer flow also starts from the trigger Pipeline events: Passed (see above). Session ids and durations are on a login-only page and are not counted.
  • The instruction block. Each spec issue carries a Rules block (tests first, push the failing check, wait for red, one retry, never edit CI files). It is committed as an issue template: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/blob/main/.gitlab/issue_templates/Spec.md
  • Pipeline. .gitlab-ci.yml has stages unit, test, deploy, live, promote, prod-live and close: staging deploy, live-check:AC-n jobs, promote:production with no manual job, prod-live-check:AC-n jobs. https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/blob/main/.gitlab-ci.yml
  • The Closer. Plain Python in Clean Architecture: domain/ (standard library only), application/, adapters/ (GitLab REST v4). It reads the Run from the API, judges it against the spec (SPEC.md, numeric conditions S1 to S20, draft 7), and renders the summary. It refuses a summary claim that has no evidence and no Unknown mark.
  • Tests first. The repository has 285 unit tests, and the pipeline runs them as a CI job: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/jobs/17440198994 (285 passed in 1.19s, commit 91a382e3; some of the tests re-derive the README's numbers). Recorded API dumps of the real Runs are replayed as fixtures, and each real Run replays as Closed (or not Closed, for the failure Run).

Challenges we ran into

  • The agent cannot merge. The merge API answers 403 insufficient_scope and the /merge quick action is refused: "Quick actions merge cannot be used with AI workflows". We had defined a Closed Run as zero human acts, so Runs #4 and #5 were first judged NOT CLOSED, and those comments are still on the issues. We then redefined the metric to "at most one merge approval", wrote the reason in SPEC.md section 8, and re-judged the same recorded jobs.
  • Green tests are not a working site. The first real Run produced a failing check and the implementation in two commits three seconds apart, so the pipeline never went red. We changed the instruction to "push the failing check first and wait for red".
  • Our own parser. It swallowed the "## Rules" section into the last criterion.
  • Pages routing. A directory page returns HTTP 302, the live check read zero bytes, and the Run failed. The agent fixed it with one commit.
  • Limits. The failures in the recovery Runs were injected by us, and the retry was requested by an @mention comment. A person assigns the issue to start the flow. Another project adds the Closer with one include line (a template in the repository, not run on another project yet), and the README carries a table of human acts per Run measured from the recorded Runs by a script, against the first real Run, where four steps were left to a person. The Closer is now the CI job close:summary. With the CI variable CLOSER_TOKEN it posts on the issue; without it (the case in the showcase project today) it reads the public API and writes the summary to the job log and the artifact closer-summary.md, and anything that needs a login (issue comments, job logs) is Unknown, and the summary is not posted to the issue. Its first real run on Run #11 failed, and we kept it in the record. In the merge pipeline 2973743971 the job stopped with HTTP 401: 401 Unauthorized on an anonymous read of the deployments API, after staging, both live checks, promote:production and the production live checks had all passed. We fixed the Closer (commit adb87a01: a refused read becomes Unknown and never a crash or a CLOSED, the summary file is always written, and the CI job token is used for the deployments, environments and merge request reads, which the GitLab docs list as accessible). In the next pipeline, https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/pipelines/2973785777, every job passed, and the job https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/jobs/17362765396 wrote Run: CLOSED for issue #11 with the line no CLOSER_TOKEN set; this summary was not posted to the issue (log and artifact only). The summaries on the five earlier recorded Runs (#4, #5, #7, #9, #10) were posted by the same code run from the owner's machine before the job existed. Of nine spec Runs (#4 to #12), seven are Closed Runs (issues #4, #5, #7, #9, #10, #11, #12), one was refused by design (failure injection, issue #6) and one was abandoned unmerged (issue #8, the planted trap did not fire).

Accomplishments that we're proud of

  • Human effort, measured by script from the seven recorded Runs (python -m infrastructure.measure_cli, table in the README): 15 human acts in all, mean 2.1 per Run (assign the issue, approve the merge; Run #9 has one @mention retry on top), against 4 steps left to a person in our first real Duo run. Waiting for the approval was 0:55 to 2:37 per Run.
  • 7 Closed Runs, each with every criterion verified live: issues #4, #5, #7, #9, #10, #11, #12 (the CI job close:summary judged #11 and #12).
  • Automatic promotion to production with no manual job, shown on a real Run: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43371209/showcase/-/jobs/17321483648
  • A Run where promotion did not happen because staging failed: pipeline 2968248261 above.
  • A recovery with one agent fix commit and no human code.
  • Issue created to production verified live (python -m infrastructure.measure_cli --end-to-end, mm:ss): #10: 4:15 with 2 human acts; #11: 11:49 with 2 human acts; #12: 13:28 with 2 human acts, each including the merge approval. Baseline (first real Duo Run, issue #3): 4 steps left to a person and no production check.

What we learned

  • Duo reads every script and list in the repository and satisfies them, so a trap must live where the repository cannot show it, such as the deployment environment's routing.
  • "The agent cannot merge" is a safety gate, not a defect. Counting that one approval honestly is better than claiming zero.
  • The check that matters is the one the agent cannot edit: the CI files and live-check script are owned by the repository owner, and none of the merged agent merge requests changed them.

What's next

Date Milestone Target number
2026-11-16 Keep the repository public and the production site live until winners are announced 100% of the live checks still pass on main
2026-11-30 Run close:summary on at least 5 more spec Runs, each judged in CI (as of 2026-10-23, 2 Runs: #11, #12) CI-judged Runs at least 6 in total, manual Closer runs 0
2026-12-15 Use it on two more repositories at least 10 more Runs, at least 3 repositories, closed rate reported with the failures included

Resources: one person, zero paid services. We do not claim cost savings; we have not measured any. For the size of the problem only: in the 2025 DORA State of AI-assisted Software Development survey, 78.7% of respondents said restoring service after a change that degrades production generally takes one hour or more, and 63.8% said more than 8% of their changes to production degrade service and then need remediation such as a hotfix, rollback or patch (our sums of the distributions in Figures 13 and 14, services.google.com/fh/files/misc/2025_state_of_ai_assisted_software_development.pdf). We do not claim this project lowers either number.

Built With

  • ffmpeg
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • gitlab-pages
  • gitlab-rest-api
  • mermaid
  • pytest
  • python
Share this project:

Updates

Submission history