Night Shift

• 

  A change that passed every test and security scan, caught by the zero-traffic canary before any user saw it.

• 

  Night Shift: the on-call agent that knows when to wake you. Built on GitLab Duo Agent Platform and Google Cloud Run.

• 

  The bad version got 0% of traffic. Night Shift tested it directly, saw 507 errors, and blocked it. Users stayed on the old version.

• 

  Same agent, two incidents: confident, so it fixed it; unsure, so it handed it to a human.

• 

  Tonight's real run: one issue in, Duo agents built, reviewed and fixed it, and it shipped. Every human touch is logged.

• 

  The autonomy gate: an agent's change only merges on its own if it's small, touches no protected files and has no security findings.

Inspiration

Inspiration

Big companies have people on call at night, plus canary releases, playbooks and audit trails. A two-person team ships at 5pm and finds out at 9am that the app was down all night. Writing code got fast. Nobody is watching what happens after it ships.

What it does

Night Shift is an on-call engineer made of GitLab Duo agents, with a written policy that decides what they may do without a human.

1. Issue to merge request. A human files an issue and assigns it to the Duo Developer flow, which writes the change and opens a merge request. Duo Code Review checks it, and the Developer flow fixes whatever the reviewer finds.
2. Tests and security scans. Unit tests, SAST, secret detection, dependency scanning and container scanning run on every change.
3. Autonomy gate. policy/autonomy.json is the contract between the team and its agents. An agent's change merges on its own only if it is small, touches no protected file (the pipeline, the gate, the policy itself) and has no high or critical findings. Anything else is labelled for a human, with the reasons.
4. Canary before users. Every new version is deployed to Google Cloud Run with zero traffic, tested directly, and only then switched live.
5. Broken change = incident, not outage. If the canary fails, users stay on the old version, and an incident is handed to the Night Shift Responder, our custom two-agent Duo flow. One agent finds the commit that caused it; the other opens the smallest fix, or hands it to a human when it isn't sure.
6. Watchdog. Production is checked every 15 minutes.
7. Paper trail. Every release gets a GitLab Release, an evidence record (what shipped, who approved it, what the canary measured, the policy in force) and a summary posted back on the original issue.

It ran for real (all on 27 Oct 2026, see the pipeline history)

• Issue to production, through two agents (issue #3, MR !3, release r40). A human filed "Show click counts for short links". The Duo Developer flow built it. Duo Code Review rejected it twice for real reasons (a race condition, then an undocumented limitation), and the Developer flow fixed both. The gate cleared it, it auto-merged, passed the canary (60 calls, 0 errors), went live, and Night Shift posted the release summary on the issue. Human touches, all logged: filing the issue, re-assigning it once after a gate fix, two one-line comments passing the review to the Developer flow, and one retry of a stuck merge job.
• A bad change stopped before users saw it (incident #2, MR !1, release r17). A change passed every test but broke link creation after ten links. The canary blocked it, the responder found the exact commit, opened a fix, the gate cleared it, and it shipped. (First live run: I fixed the gate mid-run and rebased the MR once by hand.)
• Knowing when not to act (incident #1). The first canary of the night caught a real bug whose fix belonged in the protected pipeline file. The responder diagnosed it correctly, said it was unsure, and handed it to a human.

How we built it

GitLab Duo Agent Platform: a custom flow with two agents, the Duo Developer and Code Review flows, assign triggers and service accounts. GitLab CI/CD with the security scanning templates, Container Registry, Releases and pipeline schedules. Google Cloud Run tagged revisions and traffic splitting. Python and FastAPI.

Challenges we ran into

• Who wrote this change? Duo agents act under the name of the person who triggered them, so a merge request looks human-made. The gate checks that every commit is authored by an agent's service account and that every push came from it. GitLab records pushes server-side, so a faked commit email can't fool it.
• Drafts never auto-merge. Agent flows open draft merge requests, so the gate now marks a cleared agent change as ready.
• Deciding what an agent may do alone. "Hands-off" is only safe if the hands-off part has limits, so the policy file and the zero-traffic canary carry the weight, not the model's judgement.

Accomplishments that we're proud of

• It ran for real, three ways: an agent fixed a broken change on its own, an agent correctly refused to act, and a feature went from issue to production through two agents that reviewed and corrected each other.
• No user ever saw a broken version: both bad changes were stopped at 0% traffic.
• Every release leaves an audit trail.

What we learned

Agents are most useful when their authority is explicit. The interesting work wasn't the prompts. It was deciding what an agent may do alone, proving who wrote a change, and making "I'm not sure" a real answer that routes to a human.

What's next for Night Shift

Automatically passing Code Review findings to the Developer flow (removing the last human relay step), per-file autonomy levels, and paging a human by Slack for needs-human incidents.

Built With

• ai-agents
• docker
• fastapi
• gitlab
• gitlab-ci
• gitlab-duo
• google-cloud-run
• python