Safe Until

CI only tests "now". Safe Until runs Uptime Kuma at the dates it will meet, has a GitLab Duo agent fix what breaks, proves the fix at that instant, and ships it.

Comment

Inspiration

Every pipeline asks one question: does it work now? Date bugs never fail now. Azure went down on 29 February 2012. Microsoft Exchange stopped delivering mail on 1 January 2022. Let's Encrypt's root expired on a known date in 2021 and broke clients anyway. In each case, every CI run before that day was green.

Code generation is solved. Knowing whether code will still be right in March isn't. So I built the part of "life after code" that nobody tests: the calendar.

What it does

Safe Until runs Uptime Kuma (93k stars, MIT) at the instants it is going to meet over the next 400 days: every DST change across ten time zones, with the server itself in UTC, New York and Sydney. It does this under a faked clock (libfaketime), against Uptime Kuma's real scheduler and database. Each answer is checked against an oracle that reads only the wall clock.

It found a real bomb. Uptime Kuma stores a nightly maintenance window like "01:00–04:00" as "start + 3 hours":

  • When clocks go back (Europe on 25 October 2026, the US on 1 November), it ends maintenance an hour early, so alerts fire. That's #6233, open for a year.
  • When clocks go forward, it keeps maintenance open an hour too long, so real outages are silenced.

The code has been like this since April 2023, through 44 DST changes in the zones tested. The scheduler had no tests at all. Its last fix was titled "Fixing recurring maintenance start (again)".

Then the loop runs with nobody in it:

  1. A nightly schedule finds the bomb and files an issue due on the day it goes off.
  2. The Defuser, a GitLab Duo custom flow, reproduces it, fixes the cause, pins a regression test at the exact failing instant, and opens a merge request.
  3. The merge request pipeline proves the fix with no model involved: date-pinned tests, the timewarp oracle at every hazard, SAST, secret, dependency and container scanning, and the runtime's end-of-life date. A policy job refuses any change to the oracle, the probe or CI, and any test removed or skipped. If everything holds, CI merges that exact SHA.
  4. The default branch builds the image, stamps the release "Safe until YYYY-MM-DD" (the first date it would give a wrong answer), refuses to release anything that breaks within 30 days, and deploys to Cloud Run behind a canary. A failed probe rolls the canary back.
  5. It monitors the live service, posts the release and live URL on the original issue, and closes it. A public dashboard counts down to the next hazard.

The recorded run (24 October 2026, the day before Europe's clocks went back):

  • The nightly schedule filed #1 (due 25 Oct, London and Berlin) and #2 (due 1 Nov, the US).
  • The Defuser fixed #1 and opened a merge request. The pipeline rejected it. The fix passed every instant in the issue, but the 400-day timewarp found London's spring-forward on 28 March 2027 still wrong: a window that starts inside the skipped hour ended an hour late.
  • The verdict went back to the agent: the merge request was closed with the failing hazard, the instants and the command to reproduce them, and a second Defuser session started with that verdict.
  • Attempt 2 passed everything and CI merged it. Release r13, "Safe until 2027-11-28", went to Cloud Run, and both issues closed themselves with the release and the live URL.
  • Before the fix: 112 wrong answers in 7,448 probes. After: 0 in 22,344 (server in UTC, New York and Sydney).

How I built it

  • GitLab Duo Agent Platform: a custom flow (flow registry v1) with two agent components: a small model to read the issue, a large one to fix it. Toolset: repository, git, issue and MR tools, plus run_command, so it can reproduce the bug under the faked clock before and after its change. .gitlab/duo/agent-config.yml gives its sandbox Node 26, the project's dependencies and libfaketime, with a network allowlist. CI starts the flow through the Flows API, because GitLab triggers fire only on human actions.
  • GitLab CI/CD: one .gitlab-ci.yml, three shapes (nightly scan, merge request, default branch). Its stages are named after the nine DevSecOps stages, and it uses JUnit test reports, dotenv artifacts, release:, environment:, resource_group, the security templates, GitLab Pages and pipeline schedules.
  • Determinism where it matters: the hazard calendar, the probe, the oracle, the stamp, the policy, the merge gate and the monitor are plain Node scripts. The model proposes and the jobs decide.
  • Google Cloud: Cloud Run (scales to zero, max 1 instance), Artifact Registry, and Workload Identity Federation with GitLab OIDC, restricted to this project's protected refs. No keys. The image is built from source in the GitLab registry and copied to Artifact Registry by digest.
  • A demo you can't take over: every Cloud Run boot seeds a public status page, and an admin account whose password is random and never stored.

Challenges I ran into

  • Defining "correct" honestly. On fall-back night, 01:30 happens twice; on spring-forward night, 02:30 never happens. My first oracle counted those as failures, which is unfair to any scheduler. The final oracle evaluates every reasonable reading and scores a probe only when they all agree. The bug stands on unambiguous instants alone.
  • Uptime Kuma's own load order. database.js and better-auth.ts require each other. The probe has to load them in the order upstream's tests do by accident, or the database never initialises.
  • GitLab's agent sandbox. Flows can't merge or release, and one flow can't trigger another. CI does both, with a personal token pinned to the tested SHA.
  • An agent that passes the test it was given. Attempt 1 fixed every instant in its issue and still left a bomb five months out. The judge checks the whole calendar, not the issue, and the verdict loop turns "rejected" into a second attempt instead of a dead merge request. I added that loop during the run, when the first rejection showed nothing would carry the verdict back.
  • The deploy, twice. Google's google-cloud-cli:stable image had dropped curl, and Cloud Run refuses a zero-traffic revision when it creates a service. Both were environment fixes; the release that went live is the agent's fix unchanged.
  • Upstream's policy. Uptime Kuma doesn't accept AI-generated pull requests. This fork sends nothing upstream, and keeps their policy file intact in docs/UPSTREAM-AGENTS.md.

Accomplishments that I'm proud of

  • A real, open bug in a 93k-star project, reproduced at the exact minute it goes off, a day before it goes off in Europe.
  • A verdict with no model in it: an agent can fix code, but it can't grade its own fix. The first fix it wrote was rejected, and that rejection is the best evidence the gate is real.
  • Releases that carry an expiry date, like food.

What I learned

Most "flaky" date bugs aren't flaky. They are deterministic and scheduled. Nobody runs the code on the day they are scheduled for.

What's next

More hazard classes for any app: certificate expiry, leap seconds, end-of-month billing, and locales with non-Gregorian calendars. Plus a GitLab CI component, so any project can stamp its releases "safe until".

Built With

  • artifact-registry
  • docker
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • gitlab-pages
  • google-cloud-run
  • libfaketime
  • node.js
  • uptime-kuma
  • workload-identity-federation
Share this project:

Updates

Submission history