Notary: Make AI Agents Prove Their Claims

AI agents can claim 'tested and safe.' Notary makes them prove it: signed receipts, a tamper-evident ledger, human-signed approval, and a 14-check verifier that fails closed.

Comment

🔏 Notary: Make AI Agents Prove Their Claims

AI agents can say 'my change is tested and safe.' Notary makes them prove it. CI runs the proofs. A signed receipt records the result. A human approves. An independent verifier checks 14 conditions offline and fails closed when anything is missing or changed.

Built for the GitLab Transcend 'Life After Code' hackathon on GitLab Duo Agent Platform and GitLab CI/CD.

Autonomy level: Assisted. Agents propose. A human approves each consequential step.

🔗 Links

What Link
Repository (MIT) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase
Releases https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/releases
Release v0.6.0 https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/releases/v0.6.0
Onboarding issue #1 https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/work_items/1
Pipelines (latest main: 2993875825) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/pipelines
Builder MR !1 (merged) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/merge_requests/1
Fix Pipeline MR !16 (merged) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/merge_requests/16
Docs MR !22 by the Duo developer agent (merged) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/merge_requests/22
SAST resolve flow MR !26 (merged after human correction) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/merge_requests/26
Monitoring MR !27 (merged) https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/merge_requests/27
SAST false-positive session https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase/-/automate/agent-sessions/10117131

⚡ In 30 seconds

Agent claims are only text. Notary makes sure text never decides anything:

  1. CI runs the proofs and seals the result in a signed receipt (real ed25519 SSH signature).
  2. Receipts are chained in a tamper-evident ledger.
  3. A human signs an approval over the exact commit, receipt hash and digest.
  4. An independent verifier runs 14 offline checks.
  5. A flipped byte, an expired approval, a bot approving itself or an unknown signer stops the release with a stable code like N-SBOM-MISMATCH.

🧑‍⚖️ For Judges: 3 minutes

Need: Python 3.11+ and OpenSSH ssh-keygen (Windows or Linux). The core uses only the standard library. Tools and tests need PyYAML.

git clone https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43637140/showcase.git
cd showcase && pip install pytest pyyaml
Minute Command Expected
1 python tools/gate.py 14/14 checks PASS, overall PASS
2 python tools/gate.py --attack overall FAIL with N-SBOM-MISMATCH, N-CHECKSUM-MISMATCH
3 python -m pytest tests/test_gate_approval.py -q 25 cases: 1 honest passes, the rest fail with named codes
Optional python -m pytest -q all pass (367 passed, 1 skipped in main pipeline 2993875825)

The same gate runs in CI on every push (notary-gate).

💡 Inspiration

What if an agent's 'it's tested and safe' is wrong? It could be a mistake, a hidden instruction in an issue or file, or a bot approving its own work. Our answer: the deciding evidence must never be something an agent writes. The pipeline produces it and a human signs it, like a notary who checks the document, not the person's story.

🧩 What it does

Layer What happens
Claims Agent writes a machine-readable claim.json. Only structured fields are checked.
Proofs CI re-runs proofs itself (fail-first, tamper, drift, risk).
Receipt Canonical JSON with an SSHSIG (ed25519) signature. Any edit breaks it.
Ledger Hash-chained receipts with a signed tree head.
Human approval A person signs over the exact commit, receipt hash and digest.
Verifier of record 14 offline checks. The judging code never comes from the change.
Fail closed Missing or contradictory evidence stops the pipeline with a named code.
Honest limits Every receipt lists what it does NOT prove. If emptied, check 14 fails.

The flow

Issue → Planner → Builder → MR + claim file
  → MR pipeline: pytest · claim-check · mr-check · notary-gate · SAST · secrets
      FAIL → Fix Pipeline flow proposes a fix → human reviews and merges
      PASS → Brief flow posts a review note → human decides
  → SAST finding → Duo resolve / false-positive flows propose → human reviews, corrects, merges
  → Monitoring: read-only pipeline-health check → human reads the report
  → main push: notary-draft (unsigned receipt + hash)
  → Human signs approval (tools/approve.py → approve/* tag)
  → notary-seal: re-check → rebuild → sign → 14-check verify

The 14 verifier checks

# Check Example code
1 Receipt signature valid N-SIG-INVALID
2 Claim matches receipt and schema N-CLAIM-MISMATCH, N-CLAIM-SCHEMA
3 Top-level verdict and reasons verdict not PASS
4 Signer key in allowed_signers N-APPROVAL-KEY
5 Ledger entry belongs to release N-LEDGER-CHAIN
6 Ledger chain and tree head intact N-LEDGER-TRUNCATED
7 Approval present, listed key N-APPROVAL-MISSING, N-APPROVAL-SIG
8 Commit, receipt hash, expiry, nonce N-APPROVAL-COMMIT, -RECEIPT, -EXPIRED, -REUSED
9 Author is not approver N-SOD
10 SBOM matches receipt N-SBOM-MISMATCH
11 Checksums match N-CHECKSUM-MISMATCH
12 Deploy identity N-DEPLOY-IDENTITY
13 No CI or policy drift N-CI-DRIFT
14 not_proven list not empty N-PROOF-EMPTY

The separate human-approval gate has 10 checks. It rejects an unlisted signer, a wrong commit, a missing ledger entry, a digest mismatch, an expired window, a replayed nonce, an author approving their own change, a bot approver, a scope mismatch and a kill-switch freeze. 24 failing cases are tested.

🧑‍💼 Human approval points (Assisted)

Agents propose. A human approves each consequential step. No flow can merge, sign or approve, and tools/flow_lint.py enforces that in CI.

Step Agent proposes Human decides Evidence
Plan Planner posts a plan note ('plan only; no work started') Proceed or not Issue #2, session 9951002
Build Builder opens a small MR with a claim file Reviews and merges MR !1
Fix a red pipeline Fix Pipeline flow opens a fix MR Reviews and merges MR !16, session 10065680
Review Brief posts one note: changes, verdict, risk, not proven Decides MR !10, session 10015039
Security fix Duo SAST resolve flow opens a fix MR for an SSRF finding Reviewed it, corrected it (the agent added a third-party dependency; we kept its scheme allow-list and stayed on the standard library), then merged MR !26, session 10117128
Security triage Duo false-positive flow analyses an XML finding Reads the analysis (78% false positive) and decides Session 10117131
Monitoring Developer agent opens a read-only pipeline-health tool Reviews and merges MR !27, session 10117134
Docs Duo developer agent drafts a docs-only MR Marks it ready and merges MR !22
Release seal notary-draft builds an unsigned receipt; notary-seal-dryrun rehearses the seal Signs the approval and pushes a protected approve/* tag Rehearsed in CI. See Honest limits.

🔄 Post-code lifecycle coverage

Stage Agent or automation Status
Plan Planner flow ✅ Ran
Code Builder and Developer flows ✅ Ran (MR !1, MR !22)
Test and pipeline repair Fix Pipeline flow ✅ Ran (MR !16)
Review Brief flow ✅ Ran (MR !10)
Security scanning GitLab SAST and Secret Detection on every pipeline ✅ Runs
Security remediation Duo resolve and false-positive flows ✅ Fix merged (MR !26); triage analysis delivered
Compliance Receipts, ledger, 14-check verifier, approval gate ✅ Dry-run passes; real seal not run
Monitoring Developer agent's read-only pipeline-health tool ✅ Merged (MR !27), not wired into CI
Deployment Not built (target is local-canary) ❌

🛠️ How we built it

  • Python: the core is standard library only. PyYAML for the flow validator and tests. pytest.
  • Signing: ed25519 SSHSIG via ssh-keygen -Y sign, with a pure-Python verifier pinned to RFC 8032 vectors.
  • Evidence: canonical JSON, CycloneDX SBOM, SHA256SUMS, allowed_signers.
  • CI: GitLab CI/CD on shared runners, python:3.12, SAST and Secret Detection.
  • Agents: GitLab Duo Agent Platform custom flows (Planner, Builder, Fixer, Brief) plus the built-in Fix Pipeline, Developer, SAST resolve and SAST false-positive flows.

Repository map

showcase/
├── notary/  # core: canon, claim, policy, proofrunner, failfirst, tamper, drift, risk,
│            # receipt, signing, sshsig, ledger, verify, approval, gate_approval
├── tools/   # gate, approve, seal, attack_day, blind_eval, claim_check, mr_check, flow_lint, monitor
├── flows/   # planner, builder, fixer, brief
├── evals/blind/  # 20 attack + 20 honest fixtures
├── tests/ · docs/ · .gitlab-ci.yml · .gitlab/duo/triggers.yml

GitLab Duo Agent Platform flows

Flow Trigger Observed
Planner Work item created ✅ Plan, risk preview, required proofs. Session 9951002, issue #2.
Builder Mention ✅ Refused the vague issue #2 (session 9954172). Opened MR !1 for issue #3, pipeline 2981713596 green, human merged.
Fixer / Fix Pipeline Pipeline failed ✅ MR !9: pipeline 2986227230 failed, 2986228926 passed. Fix Pipeline session 10065680 produced MR !16, merged by a human.
Brief Mention / assign / reviewer ✅ Review note with a 'not proven' list and no verdict. MR !10, session 10015039.
Developer Goal from chat ✅ Drafted docs MR !22 (session 10081895). Wrote the monitoring tool, MR !27 (session 10117134). Both merged by a human.
SAST resolve Vulnerability Report ID ✅ Session 10117128 opened MR !26 for an SSRF finding. Merged after human correction.
SAST false-positive Vulnerability Report ID ✅ Session 10117131 classified an XML finding as 78% likely false positive and gave suppression guidance.

Safety design: prompts say issue and file text is data, not instructions. Toolsets are minimal. No flow can sign, approve, merge or start another flow.

CI jobs

pytest · claim-check and mr-check (MRs) · notary-gate (gate.py --attack) · notary-draft (every main push) · notary-seal-dryrun (MRs, throw-away keys) · notary-seal (protected approve/* tag) · SAST · Secret Detection

📊 Evidence

Result Number Where
Test suite 367 passed, 1 skipped Main pipeline 2993875825
Attack Day (hostile edits) 22 of 23 caught docs/ATTACK_DAY.md
Attack Day scenarios 8 of 8 matched docs/ATTACK_DAY.md
Semi-blind eval, attacks 18 of 20 caught (90%) docs/EVAL_PLAN.md
Semi-blind eval, honest changes 20 of 20 passed (0% false blocks) docs/EVAL_PLAN.md
CI seal dry-run 14 checks + approval gate passed Pipeline 2991245755, job 17480627740
Secret detection No leaks found Pipeline 2986721897
SAST 3 medium findings in the Vulnerability Report; SSRF fix merged (MR !26); XML findings mitigated by a DOCTYPE/ENTITY pre-check (MR !15) IDs 419723406, 419723407, 420973893

Semi-blind evaluation: an outside AI saw only the README and threat model and wrote 20 attack and 20 honest changes. We built the fixtures from its text. One escape (attack_001) is the documented M02 gap. The other (attack_007, prompt injection) cannot affect verdicts, because code sets verdicts, never text.

🧗 Challenges we ran into

Challenge What we did
ssh-keygen -Y verify hung on Windows Wrote a pure-Python SSHSIG verifier
Pipelines created 0 jobs (wrong Secret-Detection template name, workflow:rules) Fixed in MR !7
Pasted flows rejected by the flow schema; assign trigger unavailable Added a flow_lint rule, used mention triggers
Agents that guess The Builder refused a vague issue. We kept that as a guardrail.
The SAST agent's first SSRF fix swapped urllib for a third-party requests import, which would break our standard-library-only verifier The human reviewer kept the agent's scheme allow-list and restored urllib, then merged. This is the Assisted pattern working.
The monitoring agent could not run pytest in its sandbox It said so in its MR; our CI ran the full suite before merge
A seal dry-run failed on an MR because it judges the tip of main, which had CI and policy drift Retried after main was clean; the drift check worked as designed
Verifier gap M02 (UNKNOWN sub-proof inside an overall PASS) Documented, not quietly patched

🏆 Accomplishments

  • 🛡️ 22 of 23 hostile edits and 8 of 8 scenarios caught with named codes. We published the miss.
  • 🧪 90% attack detection with 0% false blocks.
  • 🔐 Real cryptography, no mocks.
  • 🤖 Duo flows cover plan, build, fix, review, security remediation and monitoring, and every consequential step has a human approval point.
  • 👀 The human review caught a real flaw in an agent's security fix before it reached main.
  • 📚 Every claim points to a pipeline, job, session or MR ID.

⚠️ Honest limits

  • Final seal: notary-seal needs a protected approve/* tag. The account used for this project has the Developer role, so we rehearsed the full path with notary-seal-dryrun (14 checks and the approval gate passed) and did not run a real seal. Releases v0.1.0 to v0.6.0 are manual and carry no sealed receipt.
  • Security scanner: SAST may keep listing the urlopen line because it still calls it, with the scheme check in front. We did not record a re-scan result.
  • Monitoring: it observes pipeline history only and is not wired into CI. There is no deployed service to monitor.
  • Keys: signing and approval keys are software keys. Hardware-key touch is not verifiable offline.
  • Deploy: the target is local-canary. There is no container image or cloud deployment.
  • Verifier: the known M02 gap is open.
  • Not proven by design: code correctness beyond the listed proofs, that a human read the change, platform honesty, stolen credentials, and free prose.

📘 What we learned

  • Agent text is not evidence. Only what the pipeline reproduces counts.
  • Fail closed is a design rule.
  • Attack your own work. Hostile tests found gaps normal tests missed.
  • Publish your misses. M02 and the rehearsed-only seal earn more trust than hiding them.
  • Assisted autonomy works. Agents handle plans, drafts, fixes, security proposals and briefs. Humans keep the decisions, and sometimes correct the agent.

🚀 What's next

  1. Run a real seal with a protected approve/* tag and publish a Release carrying the sealed receipt, SBOM and checksums.
  2. Close the M02 gap.
  3. Schedule the pipeline-health check and add a deployed target to monitor.
  4. Have CI re-validate agent claim files against the schema.
  5. Wire the typed receipt-hash confirmation into tools/approve.py.
  6. Hardware-key approvals and cosign keyless signing.
  7. Container digests, scan results in receipts, and a real deploy target with canary and rollback.

🧰 Built with

Python · pytest · PyYAML · GitLab CI/CD · GitLab Duo Agent Platform · GitLab SAST · GitLab Secret Detection · ed25519 · SSHSIG · OpenSSH · CycloneDX SBOM

📄 License

MIT. See LICENSE in the repository.

Built With

  • cyclonedx
  • ed25519
  • gitlab
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • gitlab-sast
  • openssh
  • pytest
  • python
  • pyyaml
  • sshsig
Share this project:

Updates

Submission history