otp-delivery-guard

• 

  The repo: CI/CD component + two GitLab Duo flows + a demo app with the real failure mode

• 

  The two custom Duo flows, alongside GitLab's own built-in Code Review flow

• 

  The scheduled pipeline: one job only — no test, no deploy — running the check on its own

• 

  The demo app's mailer: a provider rejection is logged and swallowed, the signup still "succeeds"

• 

  The review flow catching a risky sender change on a real MR, citing the earlier incident

• 

  MR !1: the fix the agent opened after human approval. Its own pipeline passed the same check.

• 

  The four-stage coverage: Review, Verify, Monitor, Respond — autonomy level Supervised

• 

  Issue #1: a real incident the Duo flow opened on its own, with evidence quoted from the job log

Inspiration

Verification emails and SMS codes fail silently. A sender domain loses verification, a provider starts rejecting a template, a carrier filters a short code, and the signup endpoint still returns 200, because most apps send these messages fire-and-forget. Uptime checks stay green. CI stays green. Nobody notices until support tickets pile up or signups quietly drop. We wanted GitLab Duo agents to close that exact gap: not "is the server up," but "did the message that matters actually arrive."

What it does

otp-delivery-guard is four pieces of DevSecOps coverage for outbound verification messages, all built on the GitLab Duo Agent Platform:

• Review - before a merge request lands, a Duo flow reads the diff and, only if it touches sender/provider configuration (a from address, an API key, a provider endpoint), posts a comment explaining exactly what could silently break and how to check it first. Stays silent on everything else - a review flow that comments on every MR is noise, not review.
• Verify - a reusable GitLab CI/CD component runs after every deploy. It rents a real inbox (or a real UK phone number) from otp-watch, makes your app send its real verification message there, and fails the pipeline if the message never arrives, even when your app logged success.
• Monitor - the same component also runs on a schedule, every 6 hours, completely independent of any deploy. Sender/provider config can go stale with zero code change - an expiring key, a broken SPF/DKIM record - and nothing else would catch that between deploys.
• Respond - when the check fails, a second Duo flow reads the failing job's log, finds the commit that broke delivery, opens an incident issue with the evidence quoted from the log, waits for a human to approve (or redirect) the fix, and opens a merge request, whose own pipeline re-runs the check as live proof the fix works.

Autonomy level: Supervised. The agents investigate, diagnose, and prepare the fix entirely on their own; a human approves before any code change actually happens.

How we built it

• templates/otp-delivery-check.yml - a GitLab CI/CD component. It starts an otp-watch check, runs your app's own send command against a real target, polls for the real message, and writes a machine-readable result (otp-watch-report.json + dotenv variables) that later jobs and the agents both read.
• flows/otp-config-review.yml and flows/otp-delivery-incident.yml - two GitLab Duo Agent Platform flows (Flow Registry v1), one AgentComponent-only review step and one multi-step triage -> classify -> report -> human-approval -> fix pipeline, using real GitLab tools (list_merge_request_diffs, get_job_logs, create_issue, create_merge_request, ...) rather than any custom API glue.
• demo-app/ - a small Node app with exactly the failure mode this project exists to catch: its mailer swallows provider rejections and reports success anyway.
• .gitlab-ci.yml wires the component in twice - once gated to normal pipelines (post-deploy verify), once gated to $CI_PIPELINE_SOURCE == "schedule" (independent monitoring) - using GitLab's config-merge behavior to add rules to a component-generated job without forking the component itself.

Challenges we ran into

• GitLab Duo flows can't use structured output (response_schema_id isn't allowed in custom flows) - we had to replace it with a plain-text fixed-format answer plus a tiny one-word classifier agent feeding the router.
• The Duo flow's fix-it job needs GitLab's own gitlab--duo-tagged instance runners, which a self-hosted project runner can never pick up - we learned the exact sequence (turn instance runners on before approving the Duo session, not after) the hard way, after a first run got stuck with no matching runner.
• Getting an agent's diagnosis to route correctly without structured output meant hardening the router against an LLM's natural drift in casing/formatting, and making every real external call (the otp-watch API) fail cleanly instead of crashing the job with a raw stack trace when the service is unreachable.

Accomplishments that we're proud of

This isn't a scripted demo. On 2026-10-08, with the demo app's sender address pointed at an unverified domain:

• The deploy pipeline's delivery check genuinely failed - a real email never arrived, because a real provider rejected the send.
• The incident flow genuinely triaged it, opened a real incident issue with the evidence, waited for a real human approval, and opened a real merge request restoring the correct sender - whose own pipeline passed the same check again, closing the loop for real.
• Separately, the review flow was tested live on a merge request that changed the sender to a different unverified domain: it correctly flagged it as risky, quoted the exact before/after values, and referenced the earlier incident as precedent - without ever being told about that specific change in advance.

Every one of those is a real GitLab issue, merge request, and pipeline run this project produced, not a mock.

What we learned

Supervised autonomy is a trust question as much as an engineering one: the most useful thing an agent can do when it's unsure isn't act, it's stop and ask - which is why the incident flow's human-approval step isn't a formality, it's the actual design. We also learned that GitLab's CI config-merge behavior (redefining a job after it's included to layer on rules) is powerful enough to extend a shared component's behavior per-caller without ever forking it - useful well beyond this project.

What's next

The component and both flows are intentionally generic - they key off "sender/provider config" and "a message that should arrive," not off OTP specifically. Any team sending password resets, 2FA codes, order confirmations, or delivery alerts through GitLab CI/CD can drop this in and get the same four-stage coverage with their own send command.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for otp-delivery-guard

Built With

• gitlab-ci-cd
• gitlab-duo-agent-platform
• node.js
• otp-watch
• resend
• yaml