Proof of Alert

GitLab MR !4 restores POST when an explicit webhook method is absent.
The candidate source pipeline passed, separate from protected build and runtime proof.
Recorded DOWN and RECOVERY recipient receipts belong to one incident and arrive in order.

The alert is useful when it arrives

A monitor turning red is an important event, but the person depending on it needs a message. Recovery matters too: a missing recovery notification can leave someone believing a service is still down. I built Proof of Alert to follow that complete journey and keep a receipt at the receiving end.

This Bring Your Own project uses the real MIT-licensed Uptime Kuma 2.5.6 application. The work added after October 5 includes a controlled webhook regression, an independent target and recipient, a witness that checks event order, and a release workflow that connects a proposed repair to the application that actually runs.

A concrete failure and repair

The demonstration deliberately changes the webhook fallback in a test copy of Kuma. The independent check can see a monitor report DOWN while the expected recipient callback is missing. That makes the problem understandable without pretending that a successful HTTP response proves delivery.

A fresh GitLab Duo Agent Platform investigation receives the relevant source and failure information and proposes restoring POST. The controller validates the complete edit, reconstructs the candidate and creates its merge request. The release controller handles the tests and deployments; Duo proposes the code change. GitLab CI tests the candidate, and a separate protected project rebuilds the complete source into a fixed image.

What happens after the build

The Hands-off release starts with one authorized request from the operator. The controller stages an isolated canary and the independent services establish a healthy baseline, an actual DOWN event and a recovery event. The recipient must persist the callbacks in the correct order for the exact monitor and image. Production promotion then needs a fresh proof before the source merge can finish the release.

Google Cloud hosts Kuma on private Compute Engine resources and runs the target, recipient, witness and public review services. Firestore, Cloud Storage and signed release records keep the evidence connected. The public console is read-only, while operator actions have separate authenticated access.

The completed release shown in the film has a recorded seventeen-second production round and two durable callbacks. Visitors can open that incident and inspect the DOWN receipt followed by recovery for the same identifier. An earlier round that reached its deadline with no receipts remains visible, alongside the retained native failures. The successful record does not erase those attempts.

Challenges and lessons

I found that several different events can all look successful from one component's point of view. A monitor can detect an outage, a webhook can return and a build can pass, while the recipient still lacks the expected record. The witness therefore checks the whole order across independent services.

Another challenge was separating repair from recovery. Earlier operational failures required their own reconciliation and restoration. Those actions retain their original identities and cannot complete a failed native release. The successful release runs on its own after the operator starts it. Setup and recovery from exceptional failures remain separate operator responsibilities.

This project is for people maintaining self-hosted monitoring and the developers shipping its notification integrations. It establishes delivery to our owned HTTP recipient, without claiming that a person read an email or that an external notification vendor is always reliable. Next I would add more notification providers, interrupted-recipient cases and longer recovery exercises. The public console, MIT source, visible CI and short film expose the current end-to-end demonstration.

Try the project | MIT source and CI | Watch the demo