Inspiration
A green pipeline is useful only when an operator can explain how the recovery reached it. A proposed fix may look reasonable, but reviewers still need to connect the failure, the source revision, the approved patch and the final result. A workflow that pauses for approval can also resume in a different runner, where the original files and state no longer exist.
OpsGuardian addresses that review problem for platform engineers and maintainers responsible for GitLab CI. It focuses on one concrete configuration failure so that every recovery claim can be checked against a working example.
What it does
OpsGuardian turns an invalid service timeout into a reviewed recovery merge request. A custom GitLab Duo Agent Platform flow reads a failed pipeline, an earlier successful baseline on the same branch and configuration files at immutable commit SHAs. Duo authors a structured proposal citing the collected evidence. Deterministic policy permits only restoring the baseline timeout in demo/config.json when the supported health failure is present.
The candidate must pass five service contracts and three actual HTTP checks in disposable local processes. A complete version-2 receipt records the proposal, evidence hashes, revisions and exact patch. The flow pauses for human approval before MR publication.
After approval, a fresh Duo runner recollects the same evidence, checks receipt equality and reruns verification. It creates the recovery branch from the failed commit and opens a draft MR. An operator reviews and merges separately. The post-merge pipeline then checks the actual merge revision.
How it was built
The project combines a Python 3.11+ package, a CLI, SQLite incident/audit storage, a read-only local HTML dashboard, GitLab CI and a custom Duo flow defined in YAML. The Python runtime uses the standard library. GitLab provides the native agent session and CI execution; the model authors the proposal, while deterministic code validates evidence, policy, patch application and outcome provenance.
The receipt is the connection between the investigation and publication runners. Reverification reconstructs an incident from the referenced data rather than trusting a generated summary or assuming the old runner database survived. Application is tied to the exact failed checkout and preserves the reviewed file bytes.
The separate resolve command reads GitLab without changing remote state. It validates the merged MR, branch, parent, changed paths, candidate bytes, merge pipeline and health job before recording a local resolved outcome.
Challenges
Runner replacement made human approval a data-integrity problem: the candidate published later must match the candidate reviewed earlier. Recollecting evidence and checking the full receipt addressed that boundary.
Logs also required adversarial handling. Credential redaction needed to cover complete secret values without allowing hostile traces to stall processing. Regression checks cover long credentials, truncated private keys, prefixed Authorization headers and nested JSON. A later native Duo code review identified an additional redaction leak; it was independently reproduced, fixed and approved on rereview.
These checks are specific controls within the demonstrated workflow. They do not turn prompts, local hashes or SQLite into an absolute sandbox or a tamper-proof audit system.
Accomplishments
Cycle 03 completed a native Duo recovery on demo/audit-recovery-cycle-03. The failed health job returned HTTP 503 with timeout zero. Recovery MR !5 changed only that timeout to the known baseline value of 1500. Its recovery commit has the failed commit as its sole parent. After the operator merged, the merge-commit pipeline passed all five jobs, and its health job returned HTTP 200. The merged tree equals the recovery tree.
The later hardened implementation passed 91 unit/regression tests and all eight local CI/reviewer commands in a clean source copy. Main merge 5bc3438b passed five GitLab test/SAST jobs. These later implementation checks are separate from the historical cycle-03 publishing runner, which ran 77 tests.
The reviewer check also demonstrates refusal: confidence 0.4 produces blocked; recovery and apply return exit 2; no candidate is created for that incident and the checkout remains unchanged.
What was learned
The useful result is a recovery a reviewer can follow through its evidence, approval and final revision. Keeping native model reasoning separate from deterministic policy, execution and the offline demo makes the behavior easier to test and explain. A refusal path is part of that behavior: unsupported evidence should stop the workflow rather than invite a broader patch.
What's next
The next improvement is native post-merge finalization and a concise outcome summary in the originating issue. Additional runbooks can follow once each has explicit evidence requirements, a narrow patch policy and meaningful refusal tests.
The current scope is one timeout runbook. No application image build, hosted staging, production deployment, continuous production monitoring, Google Cloud bonus, measured time saving or environmental benefit is claimed.
Log in or sign up for Devpost to join the conversation.