Premortem

Rehearse a future incident on a merge request: GitLab Duo writes tests for the suspected causes, CI replays each at its named date or condition, and reproduced failures become regression tests.

Comment

Inspiration

A reminder feature ships in October and its tests pass. On October 31, a customer asks for a reminder “in 4 months” and the service crashes trying to build February 31. Nothing in the code changed; the calendar did.

Month ends, leap days, certificate expiry, counters outgrowing their format: these failures pass review because nothing in the diff looks wrong today. Premortem is for reviewers of merge requests that touch dates, billing, or external dependencies. Before the merge, rehearse the incident and keep a test that reproduces it.

What it does

When a merge request is created, a custom GitLab Duo flow is told the change broke in production at some later date, cause unknown. It investigates the MR and the existing code it calls. Each suspected cause becomes a pytest test with a named condition (a date replayed with libfaketime, or a parameter such as receipt volume) and the error it should produce. GitLab CI runs each test under ordinary conditions and under its condition, and Duo posts the result on the MR.

A hypothesis is PROVEN only if the test passes normally and fails under its condition with the predicted error. Everything else keeps a REJECTED or PRESENT label. After a fix, the same tests run in regression mode. What the reviewer keeps is a reproducible condition and a regression test, not a list of opinions.

Evidence-first review tools prove a suspected bug with a test that fails today. Premortem targets code that passes today and breaks only under a named later condition, and replays that moment.

The clearest run: a real upstream bug

MR !3 adds natural-language reminders on top of a historical Chronyk revision with a month-end defect. This is a retrospective run: the defect was fixed upstream in 2015, and the case was chosen from its public history.

  1. MR !3 opened at head ddf6d6be; the Duo flow starts.
  2. Duo's rehearsal tests (84076126, CI job): “in 4 months” on Oct 31 and Dec 31, 2026 is PROVEN (February 31 and April 31). Two 2038 hypotheses are REJECTED; the calls passed.
  3. Duo's report on the MR: tests, conditions, CI links.
  4. Separate repair to Chronyk's official fix (4dce000c): the same two tests at the same dates report 2/2 HOLDS.

This is one upstream defect under two conditions. The application's existing tests passed at the sweep's selected checkpoints (+1, +2, +5, +12 years), none of which fell on a month end. That is a comparison of chosen dates, not a claim that Premortem beats human review. Duo did not make the repair, and the adopted tests check that the calls stop crashing, not the exact reminder date.

How it was built

The GitLab Duo Agent Platform flow responds to MR creation, inspects code, writes rehearsal tests on a separate branch, follows GitLab CI, and posts an evidence comment. Python and pytest execute the tests. libfaketime changes each test process's clock; parameterized tests express conditions such as receipt volume. An optional target-branch replay adds origin context when the test can run there.

AI coding assistants helped with implementation, experiments, test cases, research, and these submission drafts. Prompt comparisons used local Claude Sonnet agents, separately from the recorded GitLab Duo executions. The Chronyk case was selected through public issue and fix research.

Challenges

A red test can support the wrong explanation. In a billing rehearsal, a certificate expired before the code reached a claimed leap-day failure. The gate rejected the leap-day explanation because the observed error differed from the prediction.

A passing repair also has assumptions. The billing repair's five adopted tests passed with simulated renewed-certificate settings. Duo's follow-up reproduced the failure with the shipped defaults: production still needs its settings updated.

The gate itself needed correction. A follow-up hypothesis was rejected because pytest's rewritten assertion omitted the expected exception name. A later commit fixes that matcher case; the original verdict remains recorded.

Accomplishments

The recorded workflow reaches from MR creation through Duo-generated tests, CI verdicts, an MR evidence comment, and adopted tests passing after a separate repair, with every step tied to a commit. The main demonstration uses a real upstream defect, and unsuccessful hypotheses stay visible.

What we learned

Urgent incident wording did not outperform plain review wording in a small local pilot. The value is in the replay contract (named condition, expected error, regression after the fix), not in the tone of the prompt. Matching the expected error is what separates a real cause from a test that is merely red.

Scope

  • The Chronyk run is retrospective. The bug was fixed in 2015 and the case was chosen knowing the fix. Answer descriptions were removed from the MR branch, but other branches, upstream history, and training data were not isolated. It does not show independent discovery.
  • The billing demo contains five planted conditions, described in its documentation before the Duo runs. It demonstrates the workflow, not blind discovery.
  • PROVEN establishes a replayed failure, not business correctness or complete coverage. No general AI advantage or production impact was measured.

What's next

Evaluate unseen application changes with explicit business contracts and controlled access to answer material. Improve failure matching and extend tests from absence of exceptions to exact outcomes. The intended benefit is less incident reconstruction and more reusable regression coverage.

Supporting links

Built With

  • custom-flows
  • gitlab-api
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • libfaketime
  • pytest
  • python
Share this project:

Updates

Submission history