BountyOps

A GitLab Duo agent improves a bounty-triage policy from a human-labelled oracle; GitLab carries each change from issue to production, and an API-collected ledger shows the human only merged.

Comment

Path A: Start Fresh · Autonomy: Supervised · Special prize: Most Stages Covered

Inspiration

Paid open-source bounties look like free money, but most are traps: crowded issues, rewards that never get paid, bounties only bots may claim, or work that takes ten times longer than the label suggests. Deciding which ones are worth an evening is itself work. BountyOps turns that judgement into an explicit policy, and lets an agent improve the policy without letting it grade its own homework.

What it does

The product. BountyOps scores paid engineering opportunities by expected hourly value (EHV):

$$ \text{EHV} = \frac{\text{reward} \times \text{P(accepted)} \times \text{competition} \times \text{P(paid)} - \text{cash cost}}{\text{estimated hours}} $$

Hard filters come first: no advertised reward, closed or already-rewarded issues, and bounties that exclude human contributors are rejected before any score counts. The hurdle is 60 USD/h. Production is a GitLab Pages board of 50 real bounty-labelled GitHub issues from a reproducible snapshot committed on 2026-10-16 (metadata only, issue bodies are never fetched), plus a release.json that records the commit, policy digest and wheel digest being served.

The post-code loop. A human writes one structured issue that states the policy defect, the expected decision, and which decisions may change, then assigns it to the GitLab Duo Developer flow. From there:

  1. Plan / Create: Duo edits the policy and code, adds an agent fixture, and opens a ready MR that closes the issue.
  2. Verify / Secure: unit tests, GitLab SAST and secret detection, plus governance gates loaded from a separate project that Duo cannot write to: changed-path allowlist; computed RED on the merge base and GREEN on the MR; a regression check against the human oracle, including the reason for each rejection; a cap on decision flips; a small anti-overfit lint; and a digest check that the oracle was not modified while candidate code ran. A security gate fails on High or Critical SAST findings or any secret.
  3. The single human action: the entrant reviews the green MR and merges.
  4. Package: the wheel is built and published to the GitLab package registry.
  5. Configure / Deploy: staging and production install the same wheel and read environment-scoped variables; staging reports decision drift against production.
  6. Release: a GitLab release links the wheel, production and provenance.
  7. Monitor: a job waits until production serves this exact commit and wheel digest.
  8. Govern: a provenance packet links issue → MR → commit → wheel → release → production, and a multi-project pipeline in the governance project collects the actor ledger from the GitLab API with its own read-only token. It fails if any human acted after the trigger beyond the one permitted merge, if that merge did not reach production, or if GitLab's own records show the merged MR changed a path outside the allowlist or skipped a required gate job.

The human oracle

The entrant labelled 12 real opportunities. All 12 were REJECT, the same decision the policy gave, so a decision-only oracle would have found nothing to fix. Requiring the reason exposed one case that was right by accident: ClankerNation/OpenAgents#59 advertises $7k in a format the policy could not parse, so it was rejected for "no explicit reward". Its label says Autonomus Agents Only. Had the parser been fixed first, the policy would have scored it at about 350 USD/h and recommended a bounty humans may not claim. Run 2 fixed the eligibility rule first: Duo went from the issue to a green, reviewable MR in about two minutes. Run 4 then fixed the parser, and OpenAgents#59 now shows its 7,000 USD reward and is rejected only because humans are not eligible, the reason the entrant gave.

Twelve public labels are a small regression suite, not a held-out test set; they catch regressions, they do not show that the policy generalises.

Runs

Every run has its evidence and ledger in the repository. Each ledger shows exactly one human event after the trigger: the merge.

  • Merge-path spike (#2): a --version flag. Gates passed; Duo's merge returned 403.
  • Policy 1, toy oracle (#3): watch_band 0.8 → 0.9. RED → GREEN, oracle unchanged.
  • Policy 2, real oracle (#6): eligibility labels matched by phrase. RED → GREEN; OpenAgents#59 fixed; 0 flips, 0 regressions.
  • Board 3 (#7): a summary of why the snapshot's opportunities were rejected. No decision changes; the first MR pipeline with SAST and the security gate.
  • Policy 4 (#8): rewards written as $7k. RED → GREEN (fixture REJECT → PROMOTE); OpenAgents#59 keeps only not_eligible; 0 flips.
  • Board 5 (#9): best EHV ignores hard-filter rejections. No decision changes.

How we built it

  • Python 3.12 standard library only; 43 product tests and 44 governance tests.
  • GitLab CI/CD with 8 stages, GitLab Pages, Package Registry, Releases, Environments (staging, production), environment-scoped variables, SAST, secret detection, a multi-project pipeline, and the GitLab REST and GraphQL APIs.
  • GitLab Duo Agent Platform: the Developer flow, triggered by assignment.
  • A separate governance project holds the gates, the human oracle and the ledger collector, pinned by commit.

Challenges we ran into

  • The flow could not merge. Merge and auto-merge both returned 403 insufficient_scope for the flow's token, and the hackathon role cannot create project access tokens. We claim Supervised instead of stretching Hands-off, and document the spike.
  • An agent that can edit CI can edit its own gates. The role cannot point the CI configuration at an external file, so the gates' include lives in an agent-writable file, and main accepts direct pushes from Developers. We moved the gate code and the oracle to a project the agent cannot write to and pinned it by commit. Because the include itself stays writable, the governance ledger re-checks each merged MR from GitLab's API records after it reaches production: allowlisted paths only, every required gate job green. That detects a bypassed gate; it does not prevent one.
  • A gate that would have blocked every later change. The regression gate required each MR to fix at least one known disagreement with the oracle. Run 2 fixed the only one, so every later MR would have failed. We relaxed the progress requirement, not the regression protection: later changes need not fix another disagreement, but still may not regress a known-correct case, lose a required reason, or flip more than three decisions.
  • A platform outage mid-demo. Two Duo sessions failed with HTTP 503 during a GitLab.com incident. Re-assigning the same issue would have put extra human actions in its ledger, so each retry used a fresh issue and the aborted ones are documented.
  • Scans that silently skipped MRs. GitLab's SAST and secret detection templates skip merge request pipelines unless AST_ENABLE_MR_PIPELINES is set; we found this by reading our own run's job list.
  • Retries are indistinguishable. The API does not say whether a job retry was automatic or manual, so the ledger counts any retry as human. We cut each job's artifact downloads to reduce transient failures.

What the ledger covers

From T0 until the run's first successful production deployment, the collector reads issue and MR notes and state changes, MR commits, the pipelines and jobs (with retries) of the MR and merge commits, and production deployments. It does not read project settings or audit events, so it shows who acted on these records, not that nobody touched GitLab elsewhere. For Supervised it also requires exactly one merge, by a human, that reached production.

On 2026-10-20 we re-collected all six runs from the API. Each has exactly one human event, the merge. Runs 3 to 5 pass the stricter gate; the spike and runs 1 and 2 fail it, because their MR pipelines were not scanned yet, the gap described above.

Accomplishments: finding nothing worth doing is a result

The production board rejects all 50 bounty-labelled issues in the snapshot, and says why: 21 advertise no reward, 20 score below the hurdle, 8 exclude human contributors, 1 is closed. Among those that pass the hard filters, the best scores 19.50 USD of expected value per hour, a third of the hurdle. Refusing bad work with evidence is the point.

Run 4 showed why the hard filters come first: once $7k was read, OpenAgents#59 scored 350 USD/h, and the board summary, which run 3 had defined over every opportunity, named it the best one. No decision was wrong, so the gates passed; run 5 fixed the definition through the same loop.

What we learned

  • Ask the oracle for reasons, not just decisions. All 12 labels agreed with the policy on the decision; only the reason exposed the one case that was right by accident.
  • The order of fixes matters. Fixing the reward parser before the eligibility rule would have promoted a bounty humans cannot claim.
  • Summaries are policy too. Run 4 changed no decision, yet the board summary named an ineligible bounty as the best. Gates that check decisions alone let that through.
  • Gates belong where the agent cannot write, and anything that stays writable needs an audit from the platform's own records afterwards.
  • Defaults are not coverage. The security scans looked enabled but skipped merge requests until we read the job list.
  • Claim the autonomy the records show. An API-collected ledger is a stronger claim than a description, so we claim Supervised, not Hands-off.

What's next

  • Check board summaries, not only decisions, against the human oracle, and extend drift monitoring from decisions to reasons and the summary.
  • A scheduled governance audit, so a deleted ledger trigger is noticed too.
  • Calibrate acceptance and payment probabilities from outcomes.
  • Hands-off, once a flow can merge with a scoped token.

Built With

  • github-api
  • gitlab
  • gitlab-api
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • gitlab-package-registry
  • gitlab-pages
  • gitlab-releases
  • gitlab-sast
  • graphql
  • python
  • semgrep
Share this project:

Updates

Submission history