boardcheck: an open-source tool that maintains itself

Four GitLab Duo agent flows run boardcheck's post-code life: triage, test-first fixes, security, releases and incident response. Governance-as-code limits them; humans approve what ships.

Comment

Inspiration

boardcheck is a small, MIT-licensed detection standard: it flags deceptive activity signals on AI-agent marketplaces (fake applications, seeded listings, unpaid-work traps, upfront-fee scams). Writing it was the easy part. Keeping it honest afterwards is the part that never ends. Someone has to triage reports, keep the Python library and the browser checker in agreement, answer the security scanners, cut releases and notice when production breaks. For a one-person project, that work is what kills it.

What it does

boardcheck now hands its post-code lifecycle to four GitLab Duo Agent Platform flows, with a human approving every outcome that ships:

  • boardcheck Fixer (Plan, Create): wakes when an issue is created or assigned to it. It classifies and sizes the issue. For a detector bug it writes a failing regression test from the reporter's data, then fixes both implementations and opens a merge request that closes the issue. For a marketplace report it builds a reproducible audit fixture plus a test. If the data isn't there, it asks for it and changes nothing.
  • Pipeline Doctor (Verify, Monitor): wakes on a failed pipeline. It reads the logs, names the root cause and pushes a ci-fix: to the same branch (never main), or explains a flake. When the failed job is the hourly production monitor, it opens an incident issue instead of touching code.
  • Security Remediation (Secure): wakes on a passed pipeline. It works through SAST, secret-detection and dependency findings: it compares installed and fixed versions, spots withdrawn advisories, opens fix merge requests, and files an evidence-backed analysis for anything only a Maintainer may decide.
  • Release Manager (Release, Govern): wakes when a human approves or @mentions it on a merge request. It picks the semver bump and writes release notes with a Governance section (what agents did, what humans decided). After the human merge, CI tags the release, publishes it and the package, and deploys staging. Production waits for a human click.

How we built it

  • Four custom flows in the flow-registry v1 format (.gitlab/duo/flows/): multi-component, with a conditional router in the Fixer (triage → fixer | auditor | end), triggered by Work item, Assign, Mention, Pipeline and Merge-request events.
  • Governance as code: .gitlab/duo/policy.yml lists each flow's allowed tools and a set of forbidden ones (shell access, approving or editing other people's MRs, changing vulnerability status, starting flows). A govern CI job (scripts/govern.py) fails the pipeline if a flow reaches beyond its allowlist, loses its prompt-injection guard, stops logging, has a broken router, or if the production deploy stops being manual.
  • GitLab CI covers tests on Python 3.9 to 3.12 plus a JS/Python parity test, SAST, secret detection and dependency scanning, a wheel and a container image, version and release jobs, Cloudflare Workers deploys with smoke tests, and a scheduled production monitor that takes a monitor_url input so incident drills don't touch production.

Challenges we ran into

The flows were tuned on live runs, and each failure changed the design:

  • The Security agent's first run tried to dismiss a false positive, but the project role can't change vulnerability status. It stopped after four rejected calls and reported clearly. We made dismiss/confirm forbidden by policy, and the flow now writes an analysis for a Maintainer instead.
  • GitLab's create_vulnerability_issue tool takes no description and rejected every call, so the flow switched to create_issue plus link_vulnerability_to_issue.
  • The Fixer's first run received only an issue number and couldn't resolve the project. It stopped safely, and the prompt now names the project.
  • Merge requests opened by an agent are attributed to the triggering human, and GitLab won't let authors approve their own MR. So the Release Manager also answers an explicit @mention, which is still a deliberate human decision.
  • Only human actions trigger flows, so the loop is a chain of short, explicit human hand-offs rather than one runaway agent. That's why this entry is Supervised.

Accomplishments that we're proud of

  • A real bug, found and fixed through the loop. The browser checker silently ignored AMS-007 (upfront-fee gating) while the Python library flagged it, so the public site called a fee-gated listing "clear". The Fixer wrote the failing test first, ported the check and opened MR !1. We verified independently that the new tests fail on the old code and pass on the fix.
  • An exact audit. For a marketplace report the auditor produced high_risk with AMS-001/002/004/007 on exactly the right listings, and correctly did not fire AMS-005 (median 775, cutoff 2,325).
  • A real release. v0.6.1 was prepared by the Release Manager, tagged and published by CI, and deployed to production after human approval.
  • Monitor back to Plan, rehearsed. An incident drill pointed the monitor at a dead URL. The Pipeline Doctor read the failure, changed no code and opened incident #4 with every failed check.

What we learned

Least privilege works best when an agent's limits are visible and enforced in CI, not just stated in a prompt. The most useful agent behaviour was stopping well: reporting a blocked action precisely instead of improvising around it. And governance records written by agents need the same review as code. The first one called a release request an "approval", which is exactly the kind of error that erodes trust.

What's next for boardcheck

Wiring the Fixer's audits into the public audit board, a Code Review flow on every MR, and a public dashboard of agent sessions and their outcomes.

Built With

Share this project:

Updates

Submission history