SignalGuard

An incident loop on GitLab for releases that answer 200 and store nothing.

Comment

What it does

SignalGuard catches one failure: a release that answers a request with a 200 and never stores it. A raw-ingest tee records every request before Umami touches it, so the loop can name exactly which ones were lost. When a release drops some, a pipeline opens an incident and rolls back, a Duo flow writes the fix, and once the fix deploys the lost requests are put back at the time they were first sent. The incident then closes with the counts.

Every night the loop reverts each past fix and checks that the gates still block it. When the gate blocked an agent's own feature on stored traffic, the verdict went back to that agent, which revised its design until the same gate passed it on the same 500 requests.

This is Path B (Bring Your Own) at the Hands-off level. People own the gates, and in runs 2 and 3 the only human action was merging the change that caused the incident. GitLab CI and a GitLab Duo Agent Platform flow handle detection, rollback, fix, merge, deploy and recovery.

The problem, and who it's for

It's for platform and backend engineers who run an event collector, ingestion API, webhook receiver or telemetry endpoint, ship several times a week, and already use GitLab CI. The failure has public postmortems. Amplitude lost 15 hours of batch events in 2021 and asked customers to resend them. Fullstory dropped every custom event property for 3 days in 2022 after a release that had passed CI and 14 days of staging. Umami itself had the same class of bug in July 2026: it returned 200 and stored no row. Each time, a release or config change accepted data and lost it without an error.

SignalGuard is a pattern with a working template. The production loop needs four endpoints from your app; the merge request gates are still Umami-shaped.

Before and after

Before (Umami as shipped, Oct 5): GitHub Actions lints, tests, builds and pushes an image, and nothing in production checks that a release stores what it accepts. A release that answers 200 and throws the event away passes every test and deploys, and the event is gone.

After (built Oct 10 to 22 on GitLab): every request is recorded before the app touches it. A release that drops one is caught in about two minutes, rolled back, fixed by a Duo flow and merged by a guard the agent can't edit. The lost requests are replayed at the time they were sent. A copy without SignalGuard runs beside it on the same traffic. Every merge request replays the last 24 hours of stored production traffic with the original visitor grouping and timing, and every night each past fix is reverted and has to be blocked again. The loop ships as an includable CI template, and the flow is public in the AI Catalog.

How GitLab automates the post-code lifecycle, in order

Cloud Scheduler calls the app every minute, and the app starts a GitLab pipeline only when an accepted request wasn't stored. That pipeline opens a GitLab Incident listing the lost requests and the merge requests to blame, rolls Cloud Run back, and starts the recovery-engineer Duo flow through the Flows API with the lost requests in its goal.

The flow writes the fix, an HTTP property test and the incident fixture. It runs the production gate on its own fix inside its sandbox, then opens a merge request. merge_guard is a job defined on main that no merge request can edit, and it merges only when every gate passed on that exact commit and no gate file changed. Deploy is blocked unless the merge names the open incident. After deploy, the lost requests go back through the normal path at their original time and the incident closes with the counts.

On record

There were three production incidents: 3 dropped, 3 recovered, net loss 0. A fourth (#3) was caught in a candidate before it shipped. The fastest detection took 101 s from the lost event to an open incident (#6).

Two hostile fixes were refused. One of them, !19, had a fully green pipeline and was refused for touching the gate file.

One agent-written feature, built from a person's product request (!45), was blocked when the replay of yesterday's stored traffic caught double-submits it answered 200 and didn't store (48, then 97). The same agent revised it from that verdict. The same gate then passed it on the same 500 requests with 0 not stored, and merge_guard merged it with no human merge. Our own traffic script (the twin runs) sent that traffic; what the gate replays is what production actually stored.

Thirteen nightly rehearsals ran from Oct 13 to 25. Each one reverted the past fixes and was blocked at merge. At the organizers' request (issue #1), both schedules are paused during judging so the repository doesn't change, and the pipeline history stands as the record.

Human actions, per incident

  • #5 (run 1, Oct 12): three. Merging the bad change (!10), fixing CI's own replay job (an unquoted YAML comment), and rebasing the flow's MR.
  • #6 (run 2, Oct 12): one, merging the bad change (!12).
  • #7 (run 3, Oct 13): one, merging the bad change (!16). A person wrote the two hostile fix MRs (!18, !19) on purpose as labelled tests, and both were refused.
  • #3 (Oct 11): a spike set up by hand, caught on a candidate branch before anything shipped.
  • !45 (agent feature): three comments and one gate rerun (the control), with 0 commits and 0 pushes, per the receipt built from GitLab's records.

Identity and access

The hackathon gives participants no bot accounts, protected branches or protected environments, and the organizers told entrants to run API automation on a personal access token and label what's automated. merge_guard follows that. It merges with the project owner's token, held in a protected CI variable that merge request pipelines (where agent-written code runs) never see. That's why GitLab shows the owner as the merger on !20 and !45; the guard's note on each MR and its public job log record the decision. Duo flow commits carry the flow's service account through composite identity, and the flow's own account merged !13.

GitLab Duo Agent Platform features used

  • A custom flow, recovery-engineer, public in the AI Catalog (v1.5.0).
  • CI starts that flow through the Flows API, not a person (canary/incident.py line 260).
  • A flow execution sandbox with Postgres, configured in .gitlab/duo/agent-config.yml, so the agent runs the production gate on its own fix.
  • The foundational Developer Flow, started by a mention on issue #11, which produced !45.
  • Two service accounts, one per flow.

Every agent session's pipeline is public under the ?source=duo_workflow pipeline filter. GitLab limits the session pages themselves to project members.

Stages

Plan: the loop opens the Incident with lost requests, blame and a timeline. Create: the Duo flow writes the fix, a property test and the fixture. Verify: accepted-implies-stored, every past incident's requests, the last 24 hours of stored traffic, upstream suites and agent property tests. Package: candidate images go to the GitLab registry, release images to Artifact Registry. Release: merge_guard and the deploy guard. Secure: SAST and Secret Detection on every push and merge request, plus a custom semgrep rule for this bug class. Configure: Cloud Run revisions, GitLab Environments, Cloud Scheduler, Secret Manager. Monitor: the tee, push detection, the post-deploy check and the public integrity page. Govern: the agent can't edit what judges it, fixtures are append-only, and the nightly rehearsal re-checks every past fix. The README has one public link per stage.

Sustainability

The production watch used to be a */5 scheduled pipeline that ran 138 times a day. Detection is now push-based and starts a pipeline only when something was dropped. The detection loop went from 4.51 gCO2e a day to 0.93, 79% less, computed to ISO/IEC 21031:2024 (SCI) with run counts and durations measured from GitLab's public API and energy modelled. Method and boundary are in canary/SCI_METHOD.md.

Links

Built With

Share this project:

Updates

Submission history