Watermark

Autonomous AI code provenance & supply-chain governance for GitLab. Tags MR origin, blocks hallucinated packages, compiles release AI-BOMs, and provides incident tracing via Duo Agent.

Comment

Inspiration

The rapid evolution of generative AI tools—such as GitLab Duo, Claude, Copilot, and Cursor—has solved the "code authoring" phase. However, generating code at the speed of thought introduces four systemic DevSecOps vulnerabilities in the "Life After Code" era:

  1. Authorship Amnesia: Once AI generates a merge request, attribution is lost. Reviewers cannot determine whether a complex architectural change was conceived by a senior human engineer or hallucinatory agent output.
  2. "Slopsquatting" & Hallucinated Dependencies: Large Language Models frequently hallucinate non-existent package dependencies or propose subtle typosquats (e.g., proposing watermark-phase3-hallucinated-pkg-j5x9). Malicious attackers monitor LLM outputs and register these phantom package names on public registries like npm and PyPI to trigger automated remote code execution inside build pipelines.
  3. The Compliance Vacuum: Traditional SBOMs list installed packages but have zero awareness of AI contributions. Emerging compliance standards (EU AI Act, SOC 2, ISO 42001) require engineering teams to prove what percentage of a release binary was generated by autonomous agents.
  4. Forensic Disconnect in Incident Response: When a production incident or zero-day strikes, SREs and incident responders cannot trace the offending file or commit SHA back to the generative prompt, Duo agent workflow, or merge approval that let it into main.

We built Watermark to act as an autonomous, audit-grade governance layer native to GitLab that tracks who authored every change, gates AI code against live public registries, produces release-scoped AI-BOMs, and maintains forensic incident traceability.

What it does

Watermark tags every merge request with its true authoring origin, subjects AI-authored code to deterministic registry verification, blocks unsafe dependencies before merge, and preserves an immutable audit trail:

  • Automated Provenance Tagging: Classifies merge requests at creation into GitLab Duo, External AI, Human, or Mixed.
  • Slopsquatting & Registry Gate: Inspects manifest diffs (package.json, requirements.txt, pyproject.toml) and validates every newly introduced dependency against live npm (registry.npmjs.org) and PyPI (pypi.org) registries via asynchronous HTTP lookups. Non-existent packages (HTTP 404) and Levenshtein typosquats trigger an immediate merge block (Watermark::Gate-Blocked).
  • Cryptographic AI Bill of Materials (AI-BOM): Automatically compiles release-scoped AI-BOM JSON artifacts detailing exact AI authorship percentages, provenance coverage ratios, and open security flags.
  • Incident Trace Engine: Allows security responders to input any file path (e.g., package.json) or commit SHA to instantly reconstruct its origin, authoring Duo Agent session, and authorizing merge request without heuristic guesswork.
  • Strict Zero-Mock Data Contract: Every metric, table row, and decision on the console reads from genuine Git history, live registry queries, or real .watermark/ records.

Complete DevSecOps Lifecycle Automation

Watermark automates all 5 post-code lifecycle stages:

  • Create: Automated provenance tagging on MR creation via Duo Flow.
  • Verify: Head-vs-base Git diff manifest extraction for package.json, requirements.txt, and pyproject.toml.
  • Secure: Public registry validation and typosquatting distance detection.
  • Release: Deterministic merge gating—blocking hallucinated packages and clearing verified dependencies.
  • Govern: Cryptographic AI-BOM generation per release tag and reverse incident tracing from file/commit to MR case file.

How we built it

Watermark is built natively on the GitLab Duo Agent Platform under the Supervised autonomy paradigm ($4,000 category):

  • GitLab Duo Agent Platform Flows:
    • Watermark Provenance Tagger (Flow ID: 1019357): Triggered automatically on Merge request (Created). Inspects author sessions and establishes baseline provenance.
    • Watermark AI-BOM Generator (Flow ID: 1019379): Triggered via Mention or release events to synthesize commit provenance records into auditable compliance artifacts.
    • Real Execution Traces: Verified in real execution traces (Session 9968631 verifying author identity, and Session 9968899 detecting missing npm packages and enforcing the gate-blocked state).
  • Core Verification Engine: Built with pure Node.js (ESM), utilizing native fetch with strict 12-second AbortController timeouts for public npm and PyPI API queries, and native child_process for Git diff extraction.
  • CI/CD Pipeline Architecture (.gitlab-ci.yml):
    • validate_static_site: Smoke-tests all 16 required repository files and JSON evidence.
    • registry_verifier_tests: Unit tests verifying package extraction, normalization, and distance algorithms.
    • ai_bom_generator_tests: Unit tests verifying AI-BOM generation and strict non-fabrication of missing data.
    • watermark_ai_bom: Triggered on Git tags to compile release AI-BOM JSON.
    • pages: Deploys the static console and live .watermark/ records to GitLab Pages upon pushes to the default branch (main).
  • Audit Console: Engineered as a zero-cost, responsive, modern Dark Theme interface adhering to strict engineering design guidelines (flat surfaces, hairline borders, dense table-first layouts, and sharp 3px corners without pill buttons or decorative glow).

Challenges we ran into

  • Live Registry Rate Limiting & Timeouts: Ensuring live HTTP registry calls to npm and PyPI remain dependable within CI runners required implementing strict AbortController timeouts and graceful handling of indeterminate network states.
  • Multi-Ecosystem Manifest Parsing: Supporting varied dependency declaration formats across Python (requirements.txt, PEP 508, PEP 621, Poetry pyproject.toml) and Node.js (dependencies, devDependencies, peerDependencies) using pure deterministic regex parsers without heavy external dependencies.
  • The Zero-Mock Data Contract: Enforcing a strict rule against fabricated placeholder rows meant that every single page, stat counter, and status tag had to handle honest empty states until a real GitLab Duo Flow or CI pipeline generated verifiable telemetry.

Accomplishments that we're proud of

  • End-to-End Working Proof: Real merge requests (!1 through !6), with !5 catching a real deliberate fake package (watermark-phase3-hallucinated-pkg-j5x9) and locking the merge gate with real Duo workload pipeline runs (#27).
  • Comprehensive Lifecycle Coverage: Automated all 5 stages (Create, Verify, Secure, Release, Govern), directly qualifying for the $5,000 "Most Stages Covered" prize.
  • Production-Grade Dark Audit Console: Built a high-density, accessible engineering console reading live records from GitLab Pages with zero paid backend or database costs.

What we learned

  • How to structure agentic workflows on the GitLab Duo Agent Platform using Flow Registry v1 YAML specifications and project triggers.
  • The critical role of Supervised autonomy in high-stakes security gates: allowing AI agents to perform multi-step telemetry gathering and registry verification while maintaining deterministic gate enforcement and human sign-off.
  • The acute supply-chain threat of slopsquatting: how effortless it is for generative models to propose non-existent dependencies, and why automated pre-merge registry validation is essential for every team shipping AI-assisted code.

What's next for Watermark

  • Expanded Package Ecosystems: Adding registry verification for Go modules (proxy.golang.org), Rust/Cargo (crates.io), and Java/Maven Central.
  • GitLab Security Dashboard Integration: Exporting hallucinated dependency findings into GitLab's native Security & Vulnerability reports.
  • Interactive GitLab CI Component: Publishing Watermark to the official GitLab CI/CD Catalog so any engineering team can enable AI supply-chain gating by adding 3 lines to their .gitlab-ci.yml.

Built With

  • ai-agents
  • ai-governance
  • css3
  • cybersecurity
  • devsecops
  • git
  • gitlab-ci
  • gitlab-duo
  • gitlab-pages
  • html5
  • javascript
  • node.js
  • npm-registry-api
  • pypi-api
  • rest-api
  • sbom
  • supply-chain-security
Share this project:

Updates

Submission history