Inspiration
I kept running into the same gap in incident response: the fix is often the easy part. The hard part is proving that the right commit was tested, merged, deployed, verified, and tied back to the original report. A green pipeline is not the same thing as a verified production release. I wanted to see whether a GitLab Duo flow could own that entire loop, not just suggest a patch. I also wanted the repository to prove its own claims: a signed history, fail-first tests, deployment evidence, and an honest list of what is still manual or unverified.
What it does
A person files a reproducible incident as a GitLab work item. That is the only manual step on the success path. A work-item-created trigger starts the Duo flow. Intake validates the report, checks for an existing MR, and decides whether the request is actionable. The recovery component creates an incident/ branch, writes a failing regression test and an HTTP acceptance script, opens a merge request, and waits for the red pipeline. It then adds the auto-merge label, commits a small fix, and waits for green. CI takes over from there: tests, SAST, secret detection, dependency and container scans, SBOM, a digest-pinned candidate image, a Vercel staged deployment, a public-alias verification, and a 60-second health/readiness window. The release report posts a manifest with the commits, pipelines, deployment ID, acceptance results, and time to verified production, then closes the work item. Malformed, duplicate, and out-of-scope reports safe-stop with an explanatory note and no repository changes. If verification fails, rollback restores the previous Vercel deployment and registry image tag. The clean project has one verified end-to-end replay: issue #7, an unknown-SKU restock bug, closed as verified in 1,017 seconds with a generated evidence bundle in docs/evidence/incident-7/.
How we built it
- A custom GitLab Duo flow with three components: intake_validator, recovery_agent, and stop_reporter. The flow uses explicit routers and safe-stop paths, and is versioned in AI Catalog and mirrored in the repository.
- A FastAPI inventory service with synthetic SKUs and in-memory state, so the incidents are safe to reproduce.
- A GitLab CI pipeline with 85 tests, a 95.58% coverage floor, SAST, secret detection, Trivy container and Vercel output scans, pip-audit, SBOM generation, and a digest-pinned image.
- A release path that builds a candidate image, runs acceptance checks in a container, deploys the same commit to Vercel, promotes the public alias, and verifies the deployment-baked SHA and behavior.
- A proof-carrying release-evidence.json generated by the release job, not hand-written.
- A fully signed main, including merge commits. The DCO jobs check every commit with the exact entrant identity.
- Documentation and tests that try to keep the claims honest: architecture, security, runbook, impact method, schema validation, CI graph tests, rollback tests, and a lock-consistency test.
Challenges we ran into
The biggest challenge was evidence binding. On one clean replay, the flow rebased the branch after the red pipeline. The test and fix SHAs changed, and the release report could not match the old MR pipelines to the new commits. That was a real failure, not a cosmetic one. I fixed it by resolving the test SHA’s project pipelines, and the next replay passed end to end. Another challenge was keeping the gates real. It is easy to draw a pipeline that looks fail-closed while a job can still run after an unrelated failure. I made the named gates explicit dependencies of deployment and reporting, and added a CI graph test that fails if they disappear. The image-versus-production split was also uncomfortable. The candidate image is not the Vercel artifact. Rather than pretend otherwise, I made the Vercel deployment ID and baked SHA the production artifact, labeled the image as candidate provenance, and added an image-tag rollback for failed releases. DCO was another hard lesson. A clean signed history is much easier to maintain from the start than to repair later. I rebuilt the submission with a signed root, signed merge commits, and strict all-commit checks. Finally, the flow and trigger are UI-managed. They are not applied by repository CI. The repo now carries versioned copies and documents the version pinning, but it cannot recreate the live trigger from a fresh clone.
Accomplishments that we're proud of
- A clean-project replay that actually closed a live work item through the automated evidence chain.
- Seven historical incidents verified in the pre-clean project, with times between 910 and 1,207 seconds.
- Fail-first red/green evidence, generated release manifests, and tracked verification logs.
- A fully signed history with no unsigned merge commits in the submitted project.
- Safe stops that behave like a product feature rather than an error path.
- A repository that a judge can clone, install, test, and run locally without credentials.
What we learned
Evidence integrity is the product. A pipeline can be green and still prove the wrong thing. Gates need to be explicit and tested. Stage ordering is useful, but it is fragile when jobs start using needs. Configuration that lives only in a UI is not reproducible. Version it, mirror it, and be clear about which copy is authoritative. DCO and signed history are easier to keep than to fix. The clean rebuild was worth it, but I would start that way next time. Safe stops and honest limitations build more trust than a perfect-looking demo. The project is stronger because it says what it does not prove.
What's next for Incident-to-Verified-Production (IVP)
- Apply the AI Catalog flow pin in the project and keep the trigger on the current version.
- Run a measured pilot instead of relying on a modeled baseline.
- Add a real semantic review step, or keep calling the current guard a scope/path guard.
- Move to a least-privilege project token or OIDC instead of a broad API token.
- Add a trigger bootstrap script and validate against the published flow schema.
- Extend monitoring beyond the 60-second health window.
- Add more incident classes and keep the evidence chain reproducible from the clone.
- Keep the live service and repository up through the judging period.
Log in or sign up for Devpost to join the conversation.