PROVE

PROVE = Plan · Realize · Oracle · Vigil · Evidence. Don't trust the agent. PROVE it.

Google Cloud (bonus): PROVE runs on Google Cloud Run (staging previews per MR commit, baseline and production: https://prove-prod-peb3ih6guq-ew.a.run.app/), with Workload Identity Federation (no stored keys) and GCS write-once records. The deployment code (Terraform and Cloud Run deploy scripts) is in prove-control: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove-control

Inspiration

Code generation is no longer the bottleneck. Verification is. A coding agent today writes the code, writes the tests, passes its own tests and declares the work done. It grades its own homework. Teams then either put a human reviewer back on every agent MR and lose the speed, or ship on trust.

Regulated industries solved this for people long ago with separation of duties (the four-eyes principle): whoever changes a production system does not approve the change. Coding agents have nothing like it. We built it from GitLab's own primitives (projects, identities, pinned tool lists, pipelines, protected paths) and not from prompt wording.

What it does

PROVE (Plan, Realize, Oracle, Vigil, Evidence) is a delivery method on the GitLab Duo Agent Platform. A human writes an issue and assigns it to the flow's service account. From there:

1. Plan. A Planner agent (custom Duo flow prove-build) turns the issue into numbered, tagged Given/When/Then acceptance criteria and posts them as one comment. A deterministic validator (no LLM) accepts only that grammar: ASCII only, no URLs, no shell characters. The issue text is treated as untrusted data. An ambiguous issue gets one question and the label prove::halted.
2. Realize. A Builder agent in the same flow reads the code in its checkout, writes the change and one unit test per criterion ([#iid/Cn] names), and pushes the branch prove/<iid>-<slug> with git. It has no shell. The MR pipeline is defined in a project no agent belongs to. It runs unit tests, builds one image per commit, re-reads its digest, makes an SPDX SBOM, runs SAST and secret detection, a findings gate (any High or Critical fails), a policy guard (any protected path or scanner suppression fails) and a flow-policy check, then deploys a no-traffic Cloud Run preview.
3. Oracle. A separate flow in a separate project (prove-oracle) writes black-box HTTP tests from the frozen criteria and the OpenAPI contract only. Its one tool commits to one branch of its own project; it cannot read code or reach the app. CI jobs freeze the bundle by SHA-256 in write-once storage, run it against the preview and write a verdict bound to seven hashes. The oracle-gate job on the MR fails closed unless a matching pass exists.
4. Release. A release controller in prove-control, the only holder of production credentials, promotes main to Cloud Run production only on a clean pass verdict and records each promotion.
5. Vigil and Evidence (designed, not built). Frozen tests as production monitors with rollback, and a per-release traceability report.

Status on 2026-10-27 (what ran live)

• Issue #1 "Loyalty discount for returning customers" was assigned to the flow's service account and the flow was started with an @mention.
• Duo session 10110128: the Planner posted criteria C1 to C4 on the issue. The Builder implemented a 10% loyalty discount in Pricing.java (not on sale items, rounding unchanged), wrote LoyaltyDiscountTest with one test per criterion, and pushed branch prove/1-loyalty-discount-for-returning-customers, commit 11d1b09c, authored by prove-build.
• GitLab rejected the merge_request.create push option, so a human opened MR !3 from the agent's branch without changing code.
• MR !3 pipeline 2993296940: unit-test, build, attest, semgrep-sast, secret_detection, findings-gate, policy-guard, flow-policy and deploy-preview all passed. The preview returns 9.00 for a returning customer where production v0 returns 10.00. request-verification failed (the pinned kernel image predated the Oracle modules), oracle-gate was skipped, and the MR stays blocked. That is the fail-closed design working.
• The kernel image was rebuilt and re-pinned, and MR !3 was rebased. In the latest MR !3 pipeline (2994043345), request-verification passed. oracle-gate then ran and failed closed with {"ok": false, "outcome": "mismatch"}, so the MR is still blocked: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/pipelines/2994043345
• The blind Oracle has never run live. No GitLab Release exists. Production serves the hand-made genesis v0.

The nine stages

How we built it

• GitLab Duo Agent Platform custom flows (flow registry v1). prove-build has a small dispatch router plus planner and builder components. The Planner uses only GraphQL work-item tools. The Builder runs with coding_environment: full but has only file tools on its checkout and run_git_command, with no shell. Every important tool option is pinned in YAML (project path, repository URL, url: null, the halt label), so the model cannot change it.
• Policy as a checked file. .gitlab/prove-policy.yml declares each agent's tools and pins. The flow-policy CI job fails unless the flow equals the policy tool for tool and pin for pin, the policy equals an allowlist baked into the kernel image, and forbidden tools (run_command, start_flow, gitlab_api_generic, web_search and others) are absent.
• Three GitLab projects as trust boundaries. prove (the app; agents write here), prove-oracle (Oracle flow and its inbox branch) and prove-control (kernel scripts, schemas, CI definitions, Terraform; no agent is a member). The prove pipeline is only an include of prove-control/ci/main.yml.
• GitLab CI/CD. MR-only pipelines for prove/* branches. Credentialed jobs run on one digest-pinned kernel image without a checkout, so no agent-written code runs where a credential lives.
• Security scans. GitLab SAST and Secret Detection, a findings gate with no allow-list, and protected ArchUnit rules (money as BigDecimal, no I/O in the pricing core).
• Google Cloud Run. prove-staging (no-traffic previews per MR commit), prove-baseline and prove-prod. Traffic moves in exactly one function, called only by the controller.
• Workload Identity Federation. GitLab OIDC tokens map to separate GCP service accounts (preview-deployer, oracle-recorder, promoter). No long-lived keys are stored.
• GCS write-once records. Criteria snapshots, bundles, verdicts and promotions are stored as canonical JSON with create-only writes and JSON Schema validation.
• Kernel. Python 3.12, standard library plus jsonschema: 33 modules, 17 schemas and about 1,090 pytest test functions.

Challenges we ran into

• Duo flow tokens and REST. The flow's composite-identity token got 404 Project Not Found from every REST project endpoint we tried: issues, repository files, branches, commits and MRs. GraphQL worked. We moved the Planner and Builder to GraphQL work-item tools pinned to the project's full path. The Builder now edits its checkout and pushes with run_git_command. We updated the flow-policy check and its tests for the new pins.
• MR creation. GitLab rejected the merge_request.create push option with "unknown error", so the agent could push its branch but not open the MR. A human opened MR !3 and said so in its description. This is the reason the live run is supervised.
• The Oracle hits the same wall. The Oracle flow commits through REST, which returns 404 for the flow token, so it has not run live yet.
• No Maintainer role. With "Developer + AI" we could not set protected branches or a custom CI config path. The pipeline is attached by an include, policy-guard turns a prevented change into a detected one, and the weakened guarantees are written down in the README.
• A red control project. prove-control main was red from 2026-10-26 because a test read a sibling repo that CI did not have. We fixed it, rebuilt the kernel image and re-pinned it.
• Flows live in the UI, not the repo. GitLab does not read flow YAML from the repository. The YAML is the checked source of record, and the running copy is pasted by hand.

Accomplishments that we're proud of

• A Duo agent took a plain-language issue to tagged criteria, a code change, one test per criterion and a pushed commit. That commit passed every security and packaging gate and is live as a preview.
• The merge stayed blocked when verification could not run. The gate fails closed.
• Separation of duties comes from project boundaries, pinned tool lists, identities and credential placement, not from prompt wording.
• An Oracle that is blind by construction: one write tool, no read tool, no shell.
• Every guarantee we could not get on this platform is written down as a residual risk.

What we learned

• Agent safety is a credentials-and-boundaries problem first. The most useful control was a CI job that diffs the flow YAML against a pinned policy.
• Identity details decide what works. The same REST tool works for a user and returns 404 for an agent's composite identity, while GraphQL works for both.
• Test every platform assumption with a probe before building on it, and record the yes or no.

What's next

• Make the Oracle commit through a path that works for the flow token, then run the first end-to-end verified delivery (Story 5.8) and a real release.
• Agent-created MRs, so the run is hands-off from issue to merge.
• Vigil (Epic 6): production monitors from frozen tests, confirmed rollback, classified regression issues.
• Evidence (Epic 7): a per-release traceability report and a nine-stage summary on the issue.
• Hardening (Epic 8): an Oracle quality check on the baseline, a flow drift check and contract tests.

Links

• Group: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792
• prove (app, contract, Duo flow and policy): https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove
• prove-oracle (blind Oracle flow): https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove-oracle
• prove-control (kernel, CI definitions, Terraform, Cloud Run deploy code): https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove-control
• Issue #1 with the agent's criteria: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/work_items/1
• Agent commit 11d1b09c: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/commit/11d1b09c42dc422fcbf5945e9a9344db0788f027 (rebased for the re-run as 09fe47dd, same author and content; branch: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/commits/prove/1-loyalty-discount-for-returning-customers)
• MR !3: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/merge_requests/3
• MR !3 first pipeline: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43364792/prove/-/pipelines/2993296940
• Agent change preview (no traffic): https://sha-11d1b09c---prove-staging-peb3ih6guq-ew.a.run.app/
• Production on Cloud Run (genesis v0): https://prove-prod-peb3ih6guq-ew.a.run.app/ (POST /quote)
• Public genesis promotion record: https://storage.googleapis.com/prove-hackathon-2026-records/promotions/genesis.json
• Demo video: https://youtu.be/m7NmcTEh4cs

Built With

• buildah
• gitlab-ci
• gitlab-duo
• google-cloud
• google-cloud-run
• java
• javalin
• junit
• openapi
• pytest
• python
• semgrep
• terraform
• workload-identity-federation