CarbonGate — Carbon Regression Agent

Seven performance candidates across ten MIT projects; microbenchmark speedups, not carbon savings.
Evidence-bound accounting: unknown inference costs block approval; estimates are not verified reductions.
CarbonGate: GitLab CI, a bounded Duo Flow, runtime checks and separate carbon-evidence gates.
GB forecast scheduling prototype; separate from Singapore electricity scenario assumptions.

Inspiration

Energy regressions ship silently. While functional bugs break test suites and trigger alerts, a single unoptimized loop, duplicate I/O read, or bad dependency bump can multiply a job's runtime and potentially its energy cost without failing a single assertion.

Traditional CI/CD pipelines treat correctness as a gate, but treat computational efficiency and energy as an afterthought. Furthermore, software carbon accounting is often plagued by "greenwashing"—tools either invent speculative savings or completely ignore the energy consumed by the AI models used to generate the fixes.

We built CarbonGate to treat computational and energy regressions with the exact same rigor as correctness: benchmarking every change in CI, deploying a GitLab Duo agent to locate and propose fixes, and enforcing a paired carbon accounting gate that strictly refuses to claim carbon savings it cannot prove.

What it does

CarbonGate adds an autonomous, fail-closed efficiency gate to the merge-request pipeline:

Paired CI Benchmarking: Each MR pipeline compares a deterministic reference implementation with the candidate using alternating trials. Repair validation separately records the candidate before and after the repair in explicit, provenance-bound reports.
Fail-Closed Runtime Gate: A median runtime ratio above 1.5x immediately blocks the pipeline (exit 1).
Bounded Duo Investigation: A maintainer explicitly triggers the custom GitLab Duo Flow by mentioning its service account in an issue. The agent reads the repository, reproduces the workload, runs tests and gates, and investigates waste such as repeated JSON parsing. A draft optimization MR is allowed only when the required correctness, runtime and carbon evidence passes; it must stop when that evidence is incomplete. This is not an automatic failed-pipeline trigger.
Evidence-Bound Paired Carbon Accounting: A secondary gate audits the repair against an explicit net-carbon balance. Crucially, it enforces a fail-closed rule: if hardware energy telemetry or cloud LLM inference overhead is unmeasured, the full net benefit is explicitly marked null and submission is blocked (recommend_submit=false). It never fabricates green savings.
Human-in-the-Loop (Assisted Level): The agent only opens draft MRs; final review and merge authority remain 100% in human hands.
Carbon-Aware Scheduling (Experimental): Deferrable batch workloads can be scheduled into lower-carbon grid intensity windows.

How we built it

GitLab Duo Agent Platform: Registered the Carbon Regression Guardian custom Flow in the AI Catalog and enabled its issue-mention trigger. .gitlab/duo/carbon-repair.yaml defines the bounded workflow and .gitlab/duo/agent-config.yml configures runtime setup. Duo uses the platform-configured model and execution sandbox; prompts restrict repair edits to workload.py. The separate local-model fixture runner has a narrow AST policy; it is not a general OS sandbox.
Explicit Measurement Modes: CodeCarbon provides software energy estimates on supported hosts. The local Apple Silicon adapter uses system CPU/GPU power estimates plus modeled RAM. An explicitly selected timing-only diagnostic records null energy in restricted sandboxes; it never converts runtime into assumed watts.
GitLab CI/CD Matrix Runners: Orchestrated parallel evaluation pipelines with uv (0.9.5) and Python 3.12, testing 10 pinned upstream open-source targets (codecarbon, slugify, holidays, schedule, cachetools, pydash, humanize, textdistance, more_itertools, attrs).
43-Test Robustness Suite: Developed a comprehensive regression test suite protecting against corrupted receipts, cross-run hash mismatches, timing-only misrepresentations, and unmeasured inference assumptions.

Challenges we ran into

Restricted Runner Sandboxes (/proc Isolation): In the demonstrated Duo sandbox, CodeCarbon initialization failed with psutil.NoSuchProcess under /proc isolation. We preserved that failure and explicitly ran both diagnostic comparisons in timing-only mode, where energy is null, not zero. Normal CI retained CodeCarbon mode and passed separately; we did not assume a fixed 30W draw.
The AI Energy Paradox: Generating automated repairs with LLMs consumes real cloud energy. If an AI tool claims it saved carbon without accounting for the electricity used during its own token generation, it is greenwashing. Designing an accounting engine that tracks both execution overhead and refuses to claim savings when LLM inference costs are untracked was our biggest conceptual and algorithmic challenge.
Microbenchmark Cloud Variance: Shared CI runners have variable CPU performance. We implemented alternating subprocess pairs, median ratio filtering across multiple repeats, and SHA-256 output verification to check equivalent outputs. Fixed fixtures, short trials and shared-runner noise still limit the conclusions.

Accomplishments that we're proud of

100% Green Test Suite (43/43 Tests Passing): Verified all core logic, regression gates, upstream comparators, and 14 newly added pair-accounting edge case tests.
Repair of a Deliberately Introduced Regression: In a separate teammate workspace, Duo session 10012240 removed repeated JSON parsing. The candidate/reference runtime ratio changed from 456.67x to 1.03x, and draft MR !5 was opened. This was a synthetic demonstration, not a production incident. Its earlier 2.03 g figure excluded unknown cloud inference cost and is not a complete Agent net benefit.
Public Ten-Project Cloud Validation: Pipeline 2983522035 ran eight prepared patch comparisons and two baseline-only profiles. Seven patches were performance candidates; Humanize showed no gain. CodeCarbon cold data lookup was 3.86x and cachetools TTL misses 2.41x in their specific microbenchmarks. These prepared patches were validated by CI, not autonomously generated by Duo, and the speedups are not CO2 reductions.
Academic & Scientific Honesty: Completed Duo session 10060238 on the final target with exit code 2 (blocked) and recommend_submit=false—proving that CarbonGate strictly halts when energy evidence is incomplete rather than fabricating misleading green metrics.

What we learned

True Green Software Engineering requires humility: You cannot claim software carbon reductions without measuring the full end-to-end lifecycle, including model inference and runner overhead.
Fail-closed is essential for AI agents: In an autonomous workflow, knowing when not to merge is far more important than aggressively committing unvalidated patches.
Bounded execution matters: The local fixture AST policy rejects unsupported syntax and the workers have time limits. File isolation and syntax checks do not guarantee that arbitrary generated Python is safe.

What's next for CarbonGate

Cloud Provider LLM Energy Telemetry: Integrating real-time token-level watt-hour estimates from cloud AI providers into the net carbon ledger.
Time- and Region-Matched Grid Evidence: Extend the existing GB forecast scheduler with verified execution location and observed task demand. The GB forecast experiment remains separate from Singapore electricity scenario assumptions; the two savings estimates are not added together.
Autonomous Upstream PR Suggestions: Extending CarbonGate from internal MRs to submitting upstream pull requests for inefficient open-source dependencies across the broader software ecosystem.

Reproducible evidence

This is a new hackathon prototype (Path A). No live deployment, production workload or verified realized carbon reduction is claimed. The final Duo run passed 43 tests and the runtime gate (exit 0), and correctly blocked the paired carbon decision (exit 2, recommend_submit=false, projected_net_saved_g=null).