Inspiration
CI is invisible electricity. Every merge runs a dozen jobs in a region nobody chose, at whatever time the merge happened. The power behind those machines can be several times dirtier in one place than another, and twice as dirty at 6 pm as at 3 am. Most of those jobs don’t need to run right now, and none of them need to run in that region. We wanted a pipeline that knows this, acts on it with nobody watching, and proves what it saved.
What it does Where first. A typical release on GitLab’s default runners (us-east1) is 0.631 g CO2e. Our runner in London does it for 0.093 g, 85% less. Choosing the place is the biggest saving, and it costs nothing at run time. Then when. The planner reads the grid forecast and holds a release that can wait for its cleanest half hour in the next 12 hours, bringing it to 87% below the default. Urgent work never waits: a production fix, a High or Critical security fix, or a rollback ships at once, by policy. Changes merged while a release waits ride along. One held release, r-852, carried sixteen merge requests. Every release reports its own carbon by the Green Software Foundation’s SCI method, with an uncertainty band. The GSF’s Impact Framework recomputes it and matches exactly. Every merge request carries a carbon receipt. It heals itself. A fault injected into production’s grid feed was found by the monitor, fixed by plain code and shipped with no human (#172). A Duo-diagnosed patch for a staging drill went from detection to production in 65 minutes with no human (#283). A broken release rolled back in 34 seconds. GitLab Duo reviews every change through a relay: Duo proposes, policy decides, the pipeline merges. Duo never pushes or approves. Any project can switch it on: one CI/CD Catalog component (mode: greenest, balanced or fastest), and an AI Catalog flow that opens the merge request for you. How we built it GitLab: parent and child pipelines, delayed jobs, environments, incidents, schedules, the registries, the CI/CD Catalog, security scanning, auto-merge, and OIDC to Google Cloud. GitLab Duo Agent Platform: the Developer flow through the Flows API, the AI Catalog, and an MCP server the agents can query. Google Cloud: Cloud Run for the site and the watcher, a Compute Engine runner in London that sleeps when idle, and Cloud Storage for an append-only ledger. All of it is Terraform, applied by the pipeline. Grid data: NESO (GB), Energinet (DK) and RTE (FR). Code: Python throughout; FastAPI and plain HTML for the site. Challenges we ran into Duo couldn’t run in our own project, so we built a relay. Duo’s own reviews then found at least seven security defects in that relay before we trusted it, including a spoofable account check and a carriage-return bypass of the command filter. The runner wouldn’t sleep, because the monitor ran on it. Then it slept too eagerly and killed held releases. A watcher outside GitLab fixed both. GitLab Free’s 400 hosted minutes ran out, so Terraform moved onto our own runner, with a guard against changing itself. Being honest about carbon is harder than computing it: uncertainty bands, the overhead of the checks, idle hours, and upper bounds labelled as such. Accomplishments that we’re proud of An exact match with the Green Software Foundation’s own tool, release after release. Breaking things on purpose and watching the pipeline fix them and ship the fix with nobody there, including a Duo-diagnosed patch merged through our policy gate. Using Duo where it’s strong, reading code deeply, and keeping it out of where it isn’t: policy, not a model, decides what ships. “Done” means proved: a gate reopens any issue until every acceptance criterion has evidence. What we learned
Where a pipeline runs matters far more than when. Timing still adds up, but a cleaner region is the largest single saving, and it’s free at run time. Idle machines matter too: our runner idling for an hour cost more carbon than a whole release.
What’s next for Carbon-Aware Delivery Move the runner to the cleanest region once a week of measurements settles it, spread across zones. Hourly carbon data for US regions. Publish each new version of the component to the CI/CD Catalog automatically.
Built With
- cosign
- fastapi
- gitlab-ai-catalog
- gitlab-ci
- gitlab-ci-cd-catalog
- gitlab-duo
- gitlab-duo-agent-platform
- google-cloud
- google-cloud-run
- google-compute-engine
- green-software-foundation-sci
- impact-framework
- kaniko
- mcp
- python
- sigstore
- terraform
