Provenant

AI disclosure, checked before merge and signed before deploy.

Comment

Inspiration

Ghostty's AI policy asks every outside contributor to disclose AI use: which tool, and how much it helped. In practice the maintainer still checks each disclosure by hand and takes "used Claude Code" on trust.

I wanted to know how big that gap is, so I pulled one year of Ghostty's merged pull requests:

  • 315 disclose AI use.
  • 23 link anything a reviewer could actually read.
  • Of the 18 linked sessions I could fetch, 8 never mention at least one file the PR changed.

One example is #9346. It changes 10 files, but its Amp thread names only 5 of them, and src/termio/Termio.zig is one of the missing ones. I only counted these cases. I did not re-judge the PRs. The point is simple: a reviewer can't see this gap without reading the session against the diff, and nobody has time to do that for every PR.

What it does

Provenant turns an AI-use policy into code that runs inside GitLab CI:

  1. Policy as code. .provenant/policy.yml lives on the target branch. It requires the ai:: label and an Assisted-by trailer, blocks the merge when the transcript skips a changed file, and never auto-merges AI-assisted work.
  2. Evidence and Duo review. For ai::assisted merge requests, a GitLab Duo flow reads an evidence packet (files, hunks, transcript line ids) and posts one verdict note, file by file, with citations to the transcript.
  3. The gate. provenant:gate re-checks every citation Duo made, applies the policy, signs a receipt, sets provenant:: labels and asks a human to review. If anything is missing, it blocks and names the exact path.
  4. Two keys to production. The receipt key signs on the merge request. After a human merges, provenant:attest re-checks the receipt on main and signs it with a release key that only protected pipelines can read. The deploy to Azure needs that attestation, so no attestation means no deploy.
  5. Offline verification. Anyone can check a receipt or the release bundle against the public keys with one command.

The demo MRs show each path. !5 changes two docs files, but its transcript names only one, so it is blocked. !4 is clean, and it still goes to a person. !9 changes the gate itself, so the previous gate judges it.

How I built it

  • Checker in Rust (provenant/checker): coverage, policy, receipts, attestation and monitoring. Receipts are signed with Ed25519 (ed25519-dalek).
  • GitLab CI pipeline: build, SAST, Secret Detection, evidence, Duo flow, gate, attest, deploy and monitor jobs. The gate is built from the target branch, so a merge request can't change the checker that judges it.
  • GitLab Duo flow for the transcript review. Only that flow's service account can supply a verdict, so a note from the MR author doesn't count.
  • Backtest in Python against the real Ghostty repo, using gh api.
  • The fork of the Ghostty website is deployed on Azure Container Apps, and a monitor job checks that the live site serves the deployed commit and that every receipt it shows still verifies.

The coverage rule is deliberately simple. For a merge request with changed files ( F ) and transcript ( T ):

$$ \text{blocked} \iff \exists f \in F \setminus E : f \text{ is not mentioned in } T $$

Here ( E ) is the set of exempt files, such as lockfiles and generated output. A file counts as mentioned if the transcript has its path, its parent directory plus file name, or a distinctive file name.

Challenges

  • Transcripts are hard to fetch. Amp thread pages are an HTML shell. Adding .md to the URL returns the real session. Even then, 8 of the 25 Amp links needed a sign-in or were gone, and all 6 linked gists were reproducers or logs, not sessions. So the gate has to read the link, not just check that it exists.
  • I couldn't trust the AI reviewer blindly. Duo can cite a line that isn't there, so the gate re-checks every quoted line against the transcript before it accepts the verdict.
  • Limited permissions. The hackathon project only gives Developer access, so I couldn't turn on "Pipelines must succeed". That is why the main-branch attestation became the real guard: an unreviewed change can't reach production without it.
  • Noisy scanners. The dependency scanner reports the whole site, not the merge request, so every docs change would be blocked. I kept those counts on the receipt but turned dependency blocking off. A high SAST finding or any leaked secret still blocks.

What I learned

  • A file mention is a floor, not a verdict. #10179 would be blocked even though the author says the code was written by hand. That's why Provenant pairs the strict check with a Duo review and always sends AI work to a person.
  • Signing is easy. Deciding exactly what a signature means is the hard part. A Provenant receipt proves which checks ran on one commit and what they found. It does not prove the transcript is true, and the project docs say so plainly.
  • Running real data early changed the design. My planning guess was 40 to 150 disclosures, and the real number was 315.

Built With

Share this project:

Updates

Submission history