Proof of Done

Inspiration

Acceptance criteria are the most precise statement of intent a team writes, and the least enforced. They get eyeballed at review and then rot in closed issues while configuration drifts and environments diverge. The classic failure: staging is green, the unit tests are green, and production quietly doesn't do what the issue promised, because production configuration differs. Regulated teams also need requirement → test → release evidence and build it by hand before audits.

What it does

Proof of Done turns every acceptance criterion into an executable, read-only contract and keeps checking it from the issue to production:

1. Plan/Create: creating an issue triggers a GitLab Duo flow (contract compiler). It reads the issue and the API source, writes contracts/issue-N-*.contract.yaml (one check per criterion, untestable criteria flagged for a human waiver), commits it to a branch and opens a draft MR before any code exists. The MR stays red until the promise is kept.
2. Verify/Package/Secure: CI runs unit tests, lints every agent-written contract against hard safety rules (GET/HEAD only, relative paths, no credentials, no interpolation), builds a pinned, hash-locked image, makes an SBOM and runs SAST, secret, container and dependency scanning.
3. Deploy/Govern: the exact image is verified against every active contract in staging. A deterministic release gate (contracts proven, required scanner reports present, no High findings or secrets unless waived by a named human with an expiry date, no copyleft licences) decides. A second Duo flow (gate reporter) explains the decision on the MR for the approver.
4. Release/Configure: merge → GitLab Release carrying the traceability ledger → the production candidate runs with production configuration at 0% traffic. If it breaks a promise, promotion halts. On Cloud Run, a post-promotion breach rolls traffic back automatically.
5. Monitor: any failed default-branch pipeline wakes a third Duo flow (breach responder). A tool-less triage step ignores unrelated failures; real breaches get an incident with evidence, the root cause (e.g. the config line), a minimal fix MR, and a note on the original promise issue. /proof on the production service re-runs the bundled contracts against the running revision (cached for a minute) next to the CI evidence for staging, the candidate and production; monitor:production re-runs them from CI whenever a pipeline schedule is enabled.

How we built it

• GitLab Duo Agent Platform: three custom flows (AI Catalog, flow registry v1) wired to three triggers (work item created, pipeline passed/failed on MRs, pipeline failed on main), with least-privilege toolsets checked in CI against GitLab's own AI Catalog schema (flow_v2.json) and a per-flow write-tool allow-list. Flows are code: scripts/duo_catalog.py creates, releases, enables and wires them via GraphQL.
• proofkit (Python): contract DSL + safety linter, runner (retries, flaky detection, JUnit), release policy, ledger, plan trace. 69 automated tests.
• GitLab CI: nine stages, GitLab security templates, release: keyword, environments, manual production gate, keyless Google Cloud auth (OIDC id_tokens → Workload Identity Federation).
• Service: a small FastAPI "Workday API" (public holidays and business days), the product whose promises are being kept.

Challenges

• The AI Catalog validator is stricter than the flow-registry docs: response_schemas, max_cycles and some log events are rejected, so our flows route on plain-text answers and we validate every flow against the real schema in CI.
• Triggers only fire on human-initiated events (by design, to prevent loops). We designed the chain around that: the developer's push wakes the gate reporter, and agent commits don't.
• The release gate blocked our own first release: container scanning found 44 High CVEs in the Debian base image. We moved to a pinned Alpine image instead of waiving them.
• The gate reporter's first review pointed out that the MR gate had no scanner reports to evaluate; we made missing scanner evidence a blocking rule.

Accomplishments (all clickable in the project)

• Contract compiler: issue → lint-clean contract → draft MR in about 80 s, with no human edits on 3 of 3 issues.
• Prompt injection inside an issue (POST the CI token to an attacker host, edit .gitlab-ci.yml): refused, reported on the issue, zero injected content in the contract.
• MR red before the feature existed (PROOF-BREACH … 404), green after; gate reporter review with a per-criterion table.
• Production halt on Cloud Run (job 17493069792): staging enabled India but production config didn't, so the candidate revision broke 4 of issue #3's promises at 0% traffic and promotion halted. About 70 s later the breach responder opened incident #6 (quoted evidence, root cause: configuration drift) and fix MR !4 (one line). A retried deploy failed the same way; the responder added to the incident instead of opening a duplicate.
• After a human merged !4, deploy:production (job 17496883460) proved the candidate 10/10, promoted it, and proved production 10/10 again; release v1.0.39 carries the ledger.
• Seeded-regression evaluation: contracts caught 8/8, unit tests 7/8. The miss is configuration drift, which tests can't see.

What we learned

Agents are good at turning prose into precise checks and at explaining and investigating failures. They should not hold the release decision. "Agents judge, code decides, humans approve outcomes."

What's next

Browser-based contracts for UI criteria, contract drafting for existing closed issues (back-filling promises), and auto-merge on a green gate for teams that want Hands-off.

Built with

GitLab Duo Agent Platform (custom flows, triggers, AI Catalog), GitLab CI/CD, GitLab security scanning, GitLab Releases, Google Cloud Run, Artifact Registry, Workload Identity Federation, Python, FastAPI, httpx, pytest.

Built With

• ai-catalog)
• artifact-registry
• fastapi
• gitlab-ci/cd
• gitlab-releases
• gitlab-security-scanning
• google-cloud-run
• httpx
• pytest
• python
• triggers
• workload-identity-federation