CrewOps: Second Pair of Eyes

CrewOps makes two people agree before a roster changes. Now its code gets the same rule: GitLab rates each change's risk, a Duo flow briefs the approver, and risky merges wait for sign-off.

Comment

Inspiration

CrewOps is my crew-control app for airlines. When a pilot calls in sick, it works out who can legally fly instead, and nothing goes onto the roster until a second person approves it. That rule is there because a wrong answer here isn't a cosmetic bug: it's a crew member flying without enough rest.

While I was working on it, I noticed the rule only protected the roster. The code that decides what "enough rest" means could still be changed and merged by one person on their own, and that person was me. In a real team that's exactly the change you'd want a second pair of eyes on. So I asked GitLab to apply the same four-eyes rule to the code itself.

What it does

Every merge request goes through four steps.

1. Change analysis. A script reads the imports across the codebase and works out what the change can actually reach, not just which files it touched. It rates the risk using plain-text rules in release/risk.toml:

The change touches Risk Must sign
The legality rules (duty, rest, flight time) or the approval and sign-in code critical ops approver and an engineer
Anything that reaches the legality engine through imports high ops approver and an engineer
How a cover plan is written to the roster high ops approver
AWS resources high an engineer
The assistant, Alexa, other code, the release checks medium nobody (review welcome)
Screens, Fire TV layout, docs low nobody

It also takes a SHA-256 fingerprint of the exact commit being judged.

2. Only the tests that matter. The pipeline runs just the tests the change can reach. A docs change runs none. A change to the Alexa skill ran 3 of the 56 test files. A change to seed data or dependencies still runs everything, and main always runs the full suite.

3. The Change Analyst. This is a custom GitLab Duo flow that starts by itself when a merge request opens. It reads the diff and the pipeline's analysis, then posts a brief for whoever has to approve: what changes for controllers and approvers, what the change reaches, and what to check before saying yes. It's written in operations language, so an ops lead can follow it without reading code.

4. The release gate. High and critical merge requests are held until every role the analysis asks for has approved. Each approver has to be a different person, none of them the author, and the approvals have to be for the commit that was analysed. If the gate can't check, it stays shut. With "Pipelines must succeed" turned on, a shut gate blocks the merge.

After the merge, main runs the full suite and then a read-only check against the live site. Public pages have to answer, and endpoints that need sign-in have to keep refusing anonymous callers. If the live feed ever answered an anonymous request with a 200, that counts as a failure, because it means a release opened something that was meant to be closed.

How I built it

The release checks are plain Python with no extra dependencies, in release/:

  • impact.py builds the import graph with ast and walks it backwards from the changed files.
  • analysis.py applies the risk rules and takes the fingerprint.
  • gate.py reads the merge request's approvals through the GitLab API, using a read-only token.
  • smoke.py runs the post-release checks.

There are 34 tests for these alone. The pipeline lives in .gitlab-ci.yml and runs on GitLab's shared runners with uv.

The Change Analyst is a flow-registry YAML file on the Duo Agent Platform. It's triggered by "Merge request created", and it runs again if you add its service account as a reviewer. I kept it supervised on purpose. Its tools can read the merge request, the diff, the pipeline's analysis and the repository, and they can post one note. It has no tool to approve, merge, label or push. It also copies the risk level from the pipeline and never decides it. So the script decides, the AI explains, and a person approves.

Challenges I ran into

The first brief the Change Analyst ever wrote found a hole in my own design. The merge request that added the checks also changed the risk rules, and the pipeline was reading the rules from the merge request itself. That meant a change could lower its own bar and walk straight through the gate. I fixed it so every merge request is judged by the rules on its target branch, never the ones it brings. The very first merge request, before any rules existed on main, says so openly in its analysis.

Test selection helps less than I'd hoped for changes near the core. A docs or Alexa change finishes in under a minute, but a change in the legality engine reaches 33 of the 56 test files and took about 23 minutes. The savings are real at the edges of the system, not at its centre, and I'd rather say that than oversell it.

I'm working alone, so there's no second person to play the ops approver. The gate shows that honestly: the legality change in the demo stays blocked with "nobody is named as ops approver yet". Naming someone in risk.toml unblocks it, and that change has to go through its own reviewed merge request.

The flow and the pipeline start at the same moment, so the brief could in theory arrive before the analysis is ready. The flow is told to say so and wait for a re-run. In practice the analysis job finishes in about 30 to 40 seconds, and the brief has always arrived after it.

Accomplishments that I'm proud of

The Change Analyst caught a real flaw in the system that runs it, in plain words, on its first run. I think that's the best case for having it at all.

The split between script and model also held up. Every risk level, sign-off list and fingerprint in the demo came from deterministic code. The model only explained them, and the gate never asked it anything.

What I learned

The risky part of a change isn't always the file it touches. It's what that file reaches. Reading the import graph made "this harmless-looking edit actually touches the legality engine" something the pipeline can show, not something a reviewer has to remember.

I also learned that an agent is easiest to trust when its job is narrow and its tools match that job.

What's next for CrewOps: Second Pair of Eyes

  • Report the gate as an external status check, so it re-opens the moment someone approves, without re-running the pipeline.
  • Name real ops approvers and mirror them in CODEOWNERS.
  • A second flow that opens an incident issue, with a first diagnosis, when the post-release check fails.

Built With

Share this project:

Updates

Submission history