Nightshift

File an issue at night, find it live by morning. GitLab Duo agents plan, code, test, secure, red-team, canary and ship your change, then post the evidence on the issue

Comment

Inspiration

Most software is shipped by small teams. A two-person team writes code as fast as a large one now, because AI writes much of it. But nobody on that team owns what happens after the commit: review, security scanning, deploys, canaries, on-call. Those jobs need specialists small teams don't have, so changes either wait or ship unchecked.

I wanted to know if GitLab Duo agents could fill those seats. Not an agent that writes code, but a night shift that takes an issue all the way to production safely, and proves it did.

What it does

You create an issue. That is the only human step.

  1. Plan: a Duo supervisor agent reads the issue and posts a plan, or asks for what is missing (nightshift::needs-info).
  2. Create: a developer subagent implements the change and opens a merge request that closes the issue.
  3. Verify: a tester subagent strengthens the tests; a reviewer subagent checks the diff against the issue and Nightshift's rules and approves it with a label.
  4. Secure: SAST, secret detection, dependency scanning and container scanning run on every MR. A security subagent fixes real findings, and an agent writes red-team probes aimed at the new code.
  5. Merge gate: policy as code. The MR merges itself only when it is approved, links its issue, and has no critical or high findings.
  6. Release: the image deploys to a Cloud Run staging service, a red-team suite attacks staging, then a canary takes 10% of production traffic and a judge compares its errors and p95 latency with the stable revision. It promotes or rolls back with no human.
  7. Evidence: every release posts a Release Dossier on the original issue: tests, coverage, all four scanners, red-team results, canary numbers, the carbon cost as an SCI score, and an autonomy ledger that counts every human touch.
  8. Self-heal: when the canary rolls back a bad change, the agents open an incident, find the bug and ship a fix with a regression test through the same loop.
  9. Self-repair: when an agent's own MR pipeline fails, CI wakes the agents with the failure; they fix the branch and the MR merges itself. A circuit breaker stops after three attempts and asks for a human.
  10. Control room and morning report: a live 3D view of every release at /nightshift, and a night shift report posted every morning.

How I built it

  • GitLab Duo Agent Platform: one custom flow (.gitlab/duo/flows/nightshift.yaml) in supervisor mode with four subagents (developer, tester, reviewer, security), each with its own prompt and a tool set limited to its job. Triggers: work item created, mention, pipeline failed.
  • GitLab CI/CD: the deterministic half. Agents decide; the pipeline enforces: security templates, the merge gate, staging → red team → canary → promote/rollback, the SCI job, the dossier, a scheduled watcher and the morning briefing.
  • Google Cloud: Cloud Run (scale to zero), Artifact Registry with cleanup policies, Firestore, Cloud Logging, Secret Manager. All infrastructure is Terraform. CI authenticates with Workload Identity Federation (no keys), and only protected branches can deploy, so an agent's branch can never touch production.
  • Python: the demo app (a FastAPI link shortener) and every pipeline tool, all unit-tested (385 tests).
  • Carbon: SCI per release (Green Software Foundation spec, ISO/IEC 21031) from CI job time and Cloud Run time, with Google's per-region grid data.

Challenges I ran into

  • Flows can't trigger flows, and bots can't trigger anything. The "pipeline failed" trigger ignores pipelines the flow's own service account started, which is every agent MR. So a CI job, nightshift-wake, runs only when an agent MR fails and posts a note mentioning the agent; the Mention trigger fires and the supervisor takes its repair path. Disclosure: GitLab only counts human actions as triggers, so that note is posted with my personal access token.
  • GitLab redacts secrets before agents see them. My first self-repair test put a key-shaped string in an issue to trip secret detection. The agents saw [REDACTED], and the supervisor paused to ask for the key instead of inventing one. Good agent, bad test.
  • The first rollback restored the wrong revision. It took the "previous" revision from the front of Cloud Run's traffic list, where every old canary leaves a 0% entry, and restored a build six releases old. It now picks the revision actually serving traffic.
  • A reviewer that waited for CI. The reviewer refused to approve while the pipeline ran, and the gate failed for lack of its label. Now the reviewer judges the code and the gate waits up to 15 minutes for the label.
  • An agent that correctly refused to improvise. On the first self-heal the fix had no issue to close and the supervisor had no tool to create one, so it stopped instead of working around the gate. I gave it create_issue and a rule to open an incident.

Accomplishments that I'm proud of

  • Issue to production with zero human touches. #16 was filed at 21:17 and was live 15 minutes later; its dossier says 0 human touches. #8, #9, #11 and #13 shipped the same way. 6 of 9 agent MRs needed no human at all, and the ledger names every touch on the other three.
  • Self-heal: a bad push to main hit 8.3% errors on the canary (1.7% on stable); production rolled back on its own, the agents opened incident #12 and merged a fix with a regression test (!9).
  • Self-repair: I pushed a bug onto an agent's MR on purpose (!11). Tests failed, CI woke the agents 22 seconds later, the fix landed 83 seconds after that, and the MR merged itself six minutes after my sabotage.
  • Evidence instead of claims: 20 releases recorded, each with a dossier, all nine DevSecOps stages backed by a real job or agent.

What I learned

  • Agents decide, code enforces. Every time an agent judged a hard rule (is CI green? is this safe to merge?), it hesitated or was wrong. Moving those rules into plain jobs made the agents more useful: they spend their effort on what code can't do, diagnosing a failure and writing the fix.
  • Subagents can't ask questions. A subagent only sees its supervisor's briefing. The prompts that work make the supervisor restate the issue, files and acceptance criteria in every delegation.
  • Where the carbon goes. A release costs about 0.81 gCO2e, and CI runner time dominates, not Cloud Run. The lever that mattered was running fewer pipelines.

What's next for Nightshift

  • A merge queue on main, so the bot token can be locked to protected branches.
  • Measuring LLM inference energy per agent session, once GitLab exposes it, to complete the SCI picture.
  • Packaging the flow and pipeline as a template any GitLab project can adopt.

Built With

  • artifact-registry
  • cloud-logging
  • docker
  • fastapi
  • firestore
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • gitlab-security-scanning
  • google-cloud
  • google-cloud-run
  • green-software-sci
  • python
  • secret-manager
  • terraform
  • three.js
  • uv
  • workload-identity-federation
Share this project:

Updates

Submission history