Issue-to-Production Agents

A speedscope fix took five months to reach users. On this fork a merged fix is live in minutes, and GitLab Duo agents take an issue to production within the hour, safely on a public repository.

Comment

Issue-to-Production Agents takes a GitLab issue all the way to the live website. A maintainer hands an issue to the agents, and that is the only human step: a planner agent writes the plan, GitLab Duo Developer Flow writes the code, Duo Code Review checks it and a fixer agent answers the review, and a small conductor merges and deploys it once every check has passed, then verifies the live site. If production breaks, an incident agent reverts the change. It runs on a public fork of speedscope, the open-source flame-graph viewer (MIT), whose fixes used to wait months to reach users.

One issue, start to finish (#18, a real upstream bug report: on CPUs with big and little cores, speedscope dropped every sample from the second core type; 21 Oct, UTC). 07:00 the maintainer hands the issue to the agents, the only human step → 07:02 the planner agent posts acceptance criteria, risk and a test plan → 07:03 GitLab Duo Developer Flow starts → 07:06 merge request open; tests, security scans and a browser check run → 07:12 Duo Code Review: nothing to fix → 07:14 merged → 07:18 deployed, checked live in a browser, released → 07:19 timeline posted on the issue. 18 minutes, nobody touched it. 10 changes have shipped this way (median 21 minutes). The results dashboard lists every run.

Inspiration

speedscope is the flame-graph viewer that py-spy, Austin and pyinstrument send their users to, with over 250,000 npm downloads a month and one maintainer. In May 2026 a Safari fix was merged; in October the live site still served a build from December 2025. Releasing needs a Mac and a script that waits for someone to type "yes". That is the common story of popular small open-source tools: the code gets fixed, but the fix doesn't ship.

What it does

A maintainer opens an issue, and agents take it to production:

  1. A planner agent (custom AI Catalog flow) writes acceptance criteria, a risk level and a test plan, or stops to ask for information the issue lacks.
  2. GitLab Duo Developer Flow implements the plan in a merge request, steered by AGENTS.md.
  3. The merge request pipeline runs tests, typecheck, lint, SAST, secret detection, dependency scanning (with a gate on findings the change introduces), an OSI licence policy, a review app on GitLab Pages and a Chromium check that opens real profiles.
  4. A fixer agent repairs failing pipelines; GitLab Duo Code Review reviews the green commit, with project-specific instructions, and the fixer answers its findings.
  5. The conductor merges exactly the commit that passed and that Duo reviewed. The main pipeline deploys to production, verifies the live site in Chromium, publishes the site bundle with a CycloneDX SBOM and SLSA provenance, and cuts a GitLab Release with notes.
  6. Once production is verified, the issue gets a timeline of every step and the change's footprint. A results dashboard lists every change in production.

When production breaks (on 16 October a deliberate break, #9, was reverted and verified 19 minutes after it failed, with no hands), the browser check after each deploy, or the one every 15 minutes, opens an incident and an incident responder agent reverts the change that caused it. Every day a security triage agent reads the vulnerability report, works out which vulnerable packages actually ship to users, and opens issues only for those, which a maintainer reads before handing them to the agents.

Results so far: 10 hands-off changes in production (8 from real upstream issues, 1 the security agent's own finding, 1 a follow-up from a Duo review), a median of 21 minutes from the agents being given an issue to verified production (fastest 18). In one of them (#10) Duo Code Review found two bugs in the agents' own change and the fixer agent repaired them before the merge; a deliberate production break (#9) was reverted by the incident responder and verified 19 minutes after the check failed. Runs where a maintainer stepped in are labelled "assisted" and listed apart on the dashboard, never counted as hands-off: 2 so far. Production here is this fork's own site (and its copy on Google Cloud); speedscope.app belongs to the upstream maintainer, so the agents' fixes go back as pull requests.

The agents do the judgement work, and it shows in public. On the first public change (a protobufjs upgrade the security agent found), Duo Code Review pointed out that the new test only checked that nothing threw, while the importer swallows decode errors, so it passed on the vulnerable version (as the merge request itself admitted). The fixer agent rewrote it to fail on the old version, and only then did the change ship (!1).

Who it's for

Maintainers of small, popular open-source tools, where one person holds the release keys and fixes wait months to ship. The same setup runs on any fork: ops/README.md takes it from publishing the flows to a free-tier Google Cloud VM.

What's new here

  • Agents check agents. Duo Code Review reviews the Developer Flow's work and the fixer agent answers it: it caught a test that passed without its fix (!1) and two crashes (!6) before anything merged.
  • Security work only where it ships. The triage agent turned 16 packages with Critical or High advisories into 1 issue, the one that reaches users' browsers.
  • Tested by breaking it on purpose. A deliberate production break was reverted by the responder agent, with no hands.
  • Safe on a public repository. Only the project's direct members can give the agents work, and an outsider's comment stops an issue before anything merges, with a test for every rule.

How we built it

  • GitLab Duo Agent Platform: the foundational Developer Flow and Code Review flow, plus four custom flows defined as YAML in the repository and published to the AI Catalog by a script, so the repository is their source of truth.
  • GitLab CI/CD for every check, GitLab Pages for production and per-merge-request review apps, the package registry, Releases, the vulnerability report, and OpenTofu with GitLab-managed state for the project's own labels, schedule and settings.
  • A small, dependency-free Python conductor between the agents. GitLab's flow triggers need a person's action (a pipeline started by an agent's commit cannot start the next agent), so the conductor acts as the maintainer: it starts each agent, reads its result, merges, and reports. Agents do the judgement; the conductor only the bookkeeping. That is why merge requests and labels show the maintainer's name, and its comments are signed as automated. Any step a person took during a run marks it assisted. It keeps no database: state lives in scoped labels and in its own comments, so any round can crash and the next one continues. Only the tripwire's baselines (pinned label ids, the production URL, a jobs cursor) sit in a small state directory on the VM.
  • Google Cloud: the conductor runs around the clock on a free-tier e2-micro VM with no inbound access and only an IPv6 address, from a commit the maintainer pinned, with the GitLab token in Secret Manager. A publisher on the same VM copies each smoke-tested build to a public Cloud Storage bucket, so no CI job holds a Google credential. The setup script and both services are in ops/gcp/.

Challenges we ran into

  • A public project is an attack surface for agents. Anyone can comment, open a fork merge request that says "Closes #N", or reference an issue from elsewhere, and agents read all of it. The conductor acts only on issues from the project's direct members, locks every discussion it takes on, stops an issue when an outsider wrote on it, never adopts a fork's merge request, reads its own state markers only from its own comments, and refuses to merge any change to the pipeline or the agents' configuration. An adversarial review by a separate agent found several of these holes before launch.
  • Where the token lives. Whoever can push to main decides what a pipeline runs, so a token stored in CI can be read by anyone with push rights, whatever its scope. This project's main accepts pushes from every Developer, including people who inherit access from a parent group, and participants cannot change branch protection. So here the maintainer's token never enters GitLab: the conductor runs on a Google Cloud VM the maintainer controls, from a pinned copy of the repository, and a tripwire halts it if the delivery files on main change behind its back. For projects where only maintainers can push to main, the repository includes a CI mode with the token in a protected environment; the conductor checks those settings before every run and refuses otherwise. It has run that way: in speedscope-agents/speedscope-ci, on 23 October, upstream issue #370 went from hand-over to live in 54 minutes with every conductor step a CI job (one pipeline retried by the maintainer after a runner fault, so not counted as hands-off).
  • Platform realities. During GitLab.com's 16 October outage (GitLab's incident) the production check could not run; the conductor opened incident #8, the responder's sessions failed on the same outage, and the conductor stopped and asked for a person. Since then only a check that ran and failed opens an incident, CI retries GitLab's own failures, and an agent session that dies on the platform waits for a check that really runs instead of spending its retry. In the private sandbox on 14 October, Developer Flow sessions sometimes lost their connection a minute in, and the conductor restarted them; during a GitLab.com CI incident it retried the failed jobs once instead of sending a repair agent after a problem that wasn't in the code. Duo Code Review posts its summary a second before its inline comments, so the conductor waits for the review to go quiet. In public on 15 October a package job lost its connection and the maintainer retried it by hand; the conductor now does that retry itself.
  • Our own gate had a gap, found in public. On the first public day Duo Code Review failed to start on one merge request; a maintainer requested it again, and the conductor merged eleven seconds before the review arrived, because it treated the review as finished before GitLab recorded it as submitted (we could not identify afterwards which note it acted on). On !1 the fixer's rewrite also merged after a green pipeline without a second Duo review. The run is labelled assisted, a correction sits on the issue, and the conductor now merges only once GitLab itself records the review as submitted, with a test for exactly that case.

Accomplishments that we're proud of

  • A real upstream issue (#4) and a security fix the agents found themselves (#3), live with no human step and a public timeline each (10 hands-off changes so far: #3, #4, #7, #10, #13, #14, #15, #17, #18, #19); the runs where a maintainer stepped in (#6, #12) are labelled and kept out of the figures.
  • All nine DevSecOps stages built; the stage table in the README links the public evidence for each.
  • Agents that are safe to run on a public repository, with every safety rule covered by a test.
  • A measured footprint for every change (Software Carbon Intensity, ISO/IEC 21031:2024), and a design that avoids agent work that need not happen: 15 of 16 packages with Critical or High advisories never became work because they don't ship.

What we learned

The judgement work (planning, coding, reviewing, diagnosing) belongs to agents; the bookkeeping (who may ask, what may merge, what shipped) belongs to plain, tested code. Trust boundaries matter more for agents than for people, because agents read everything they are given.

What's next

  • More of the agents' fixes upstream: two are open as pull requests already (#557, #558), and the delivery system itself could follow.
  • Run it on your own fork: ops/README.md has the steps, from publishing the flows to starting the conductor on a free-tier Google Cloud VM.

Built With

Share this project:

Updates

Submission history