DORA

DORA rolls back a bad deploy in under a minute, lets a GitLab Duo agent write the fix, and merges it only when the pipeline proves it: red on the bad commit, green on the fix.

Comment

Inspiration

A bad deploy with nobody on call can turn a small issue into a major problem before anyone even notices. I think reliability is not just about writing good code, but also making sure someone can respond when things go wrong. I want to build tools that help teams catch issues early and make systems safer for the people who depend on them.

Small teams and solo developers ship to Cloud Run with CI/CD every day, but nobody is on call. AI can already write a fix in seconds; the hour after a bad deploy is still a human's problem. That hour is exactly what the DORA metrics time to restore service and change failure rate measure. I wanted an agent that works that hour, and a pipeline that refuses to trust it without proof.

Agents write code fast. Who proves the fix?

What it does

One human action (merging a normal feature MR), and then the loop runs by itself:

  1. The main pipeline deploys the new revision as a canary taking 10% of real traffic on Cloud Run.
  2. Canary verification replays real request shapes against canary and stable and compares the 5xx rate and p95 latency with the policy. On a breach, a deterministic job rolls traffic back in under a minute (no AI in this path) and opens a GitLab incident with the failing requests, the crashing file:line from Cloud Logging, the bad and good commits and a machine-readable record.
  3. The failed pipeline triggers my custom GitLab Duo flow. Triage exits unless it's a real incident. Fixer reproduces a failing request as a regression test, writes the smallest fix and opens an MR. Reviewer, a separate agent that can't change code, posts a verdict.
  4. The fix branch runs the red-green proof: the agent's new test must fail against the bad commit's code and pass against the fix, plus SAST, secret detection, dependency and container scanning.
  5. A merge gate running on main, from main's own code and rules, checks who pushed the fix, the pipeline, paths, size, the proof, the scans and the reviewer's verdict, then merges pinned to the verified commit.
  6. The fix canaries again, is promoted, the incident's failing requests are replayed against production, a GitLab Release is cut and the incident closes itself with its timeline.

What happens when something goes wrong:

  • The fixer isn't confident (< 0.6) or needs files outside app/ and tests/: it posts needs-human: and stops.
  • The test doesn't fail on the bad commit, a scan finds a new High/Critical, or the reviewer requests changes: the gate blocks, labels needs-human and explains why.
  • A merged fix breaks production again: rollback, and the incident escalates to a human. The agent never fixes its own fix.
  • Another change lands while an incident is open: the deploy freeze holds it.

What a human still decides: merging their own feature MRs (that merge starts every run), anything labelled needs-human, and the policy, gates and flow, which are all off-limits to the agent.

How we built it

  • GitLab Duo Agent Platform: one custom flow (Flow Registry v1) with three agents (triage, fixer, reviewer) joined by conditional routers on single-token answers, so routing is deterministic. Trigger: Pipeline events → Failed. Each agent gets only the tools its job needs: triage only reads; the fixer can branch, commit and open an MR but can't comment on MRs (so it can't approve itself); the reviewer can read and post one note but can't change code. AGENTS.md carries the rules, the MR template and the review rubric.
  • GitLab CI/CD: 21 jobs across verify, package, secure, govern, deploy, monitor and release. Every deterministic step is a tested Python/Bash script (scripts/: 53 guard tests + 17 app tests).
  • GitLab features: incidents, labels as state, merge requests, environments (canary, production, console), releases, Container Registry, SAST, secret detection, dependency and container scanning, protected CI/CD variables, and the pipeline trigger API with the job token (the fix branch asks main to judge it).
  • Google Cloud: Cloud Run (canary revisions, tags, traffic splits, rollback, promotion), Artifact Registry (the exact image digest scanned in GitLab), Cloud Logging (error evidence with file:line), Workload Identity Federation (keyless CI sign-in, scoped to this project's path) and the Cloud Run Admin API (the Guard Console reads the live traffic split with a read-only identity). The Guard Console is the live link: it shows every incident and its timeline from the project's real data.

All 9 lifecycle stages:

  • Plan: incidents opened with evidence and a machine-readable record; needs-human escalations.
  • Create: the Duo fixer writes the regression test and the fix, and opens the MR.
  • Verify: unit tests, guard self-tests, flow validation, the red-green proof, the reviewer agent.
  • Package: one Docker image per commit, in GitLab Container Registry and Artifact Registry, deployed by digest.
  • Secure: SAST, secret detection, dependency and container scanning; new High/Critical findings block the merge.
  • Release: canary, promote, post-fix production check, GitLab Release with the agent's release note.
  • Configure: Cloud Run traffic splits and revision tags as code; policy as code (policy/guard-policy.yml).
  • Monitor: canary verification, Cloud Logging evidence, the incident timeline, the Guard Console.
  • Govern: merge gate on main with main's rules, protected merge token, provenance check before deploy, deploy freeze, full audit trail.

Safety model (the GitLab Security Standard): authorize every actor (per-agent toolsets, the flow token can't merge, keyless GCP) · isolate the work (fix branches never see the merge token) · verify the checks that judge the work (the gate runs from main) · release exactly what was verified (one digest, merge pinned to the verified SHA) · respond fast (rollback in under a minute, deploy freeze, escalation instead of guessing).

Challenges we ran into

  • My hackathon role is Developer: no project access token, no branch protection, no pipeline variables. Instead of asking for more access, I redesigned: the merge token became a protected variable that only main pipelines receive, and the gate moved onto main. That made it stronger, because the branch being judged can no longer change its judge.
  • Flow tokens can't merge, and triggers only fire on human actions. I turned both into features: CI merges, and the platform rule is part of the loop guard.
  • The first real run surprised me. The gate (correctly) blocked a fix for two reasons I hadn't foreseen: GitLab shows the human as the MR author (composite identity), so the gate now checks who pushed the code; and container scanning reports the base image's OS packages on every commit, so the gate now blocks only findings the fix introduces.
  • One of my own gate notes @mentioned the flow's account, and because it was posted with a human token it started the flow again. Bot notes never @mention the flow now, and the trigger fires only on pipeline events.
  • The reviewer agent rejected the fixer's first attempt: the fix also changed initials for three-word names, which no failing request needed. I tightened one rule ("every input that worked before must give the same response") and re-armed the incident. That's the one human action in this run, and it's stated on the incident.

Known limitations: canary verification replays a fixed sample of request shapes, so a bug that sample never exercises can pass; detection covers errors and latency, not wrong answers that return 200; the merge uses my own token (masked, protected), so GitLab shows me as the merger; with a Developer role main can't be set to "push: no one", so the deploy refuses commits the flow account pushes directly, which detects rather than prevents; one service in one region.

Accomplishments that we're proud of

  • It runs for real, end to end. In incident #3, the canary served a bad revision to 10% of traffic for 3 min 09 s, production was rolled back 56 s after detection, and all five failing requests were verified working in production after the fix.
  • The second fix went from the agent's MR to verified in production in 9 minutes, with nobody touching it: a one-line fix, a regression test built from a real failing request, red on the bad commit, green on the fix, reviewer approval, and 9/9 gate checks.
  • Separation of duties actually worked: the reviewer agent caught a fix that changed more than it needed to, and the gate never let it through.
  • Proof-carrying fixes: the agent has to prove the bug exists before its fix counts. CI enforces it; the agent can't just claim it.
  • All 9 DevSecOps stages touched by one loop, with real pipelines, real incidents, real releases and a live Google Cloud deployment, nothing simulated.

What we learned

We learned that automating a fix is only half the problem; proving that it actually works and is safe to deploy is just as important. Separating the agent that writes code from the systems that verify and approve it made the workflow more trustworthy. We also learned to design around real permission constraints instead of relying on access we don't have.

What's next for DORA

  • Multi-service canaries, and wrong-answer detection with contract tests (catching 200 responses with the wrong values).
  • A supervised mode with an approval step, for teams that want a human click before every merge.
  • A real bot account and "push: no one" on main in a team setting, so the provenance check becomes prevention, not just detection.
  • Track the DORA metrics themselves (time to restore, change failure rate) across many incidents in the Guard Console.

Links: repo https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/34767100/showcase · live Guard Console https://guard-console-fojieuq2iq-el.a.run.app · video https://youtu.be/IOTJu2S31-o

Built With

  • artifact-registry
  • claude
  • cloud-logging
  • docker
  • fastapi
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-cloud-run
  • python
  • workload-identity-federation
Share this project:

Updates

Submission history