Cloud Infrastructure Drift Detective

What it does

A scheduled GitLab CI pipeline compares the declared OpenTofu configuration with the live Google Cloud environment using a read-only identity. When it finds genuine out-of-band drift, it creates a redacted evidence issue and starts a GitLab Duo Agent Platform flow.

The investigator agent reads the issue, cross-checks the actual pipeline log, inspects the declared infrastructure, and makes one of three decisions:

• Adopt a low-risk live change when reverting it could reduce availability.
• Revert unsafe or unintended drift, such as disabled authentication.
• Escalate when neither revert nor adopt is safe to choose: a changed runtime identity, a plan that would delete or replace a resource, contradictory or missing evidence, or uncertainty.

For adopt and revert decisions, a remediator agent creates a minimal, reviewable merge request. GitLab CI runs formatting, validation, tests, a plan, and policy guardrails. A human reviews the outcome and triggers the manual apply job. The pipeline then verifies that no drift remains and closes the incident issue with links to the evidence.

This is intentionally a Supervised agent: it autonomously investigates and prepares safe remediation, but a human remains accountable for cloud changes.

How we built it

The agent is constrained in two layers. The first holds no matter what the model does. The second is a versioned playbook that the agent followed in every test, but which is not technically enforced.

Enforced by design (independent of the model)

• The agent cannot change cloud infrastructure. The flow never receives cloud credentials. Only pipelines on main can obtain the write identity, and the pipeline uses it only in the human-triggered manual apply job.
• The agent has no tools to merge, approve, or run pipelines.
• CI blocks unsafe desired states (for example, making the internal API public) even if an agent proposes them.
• Sensitive plan values and personal email addresses are redacted by a script before any evidence is published.

Required by the playbook (followed in all four tests)

• Work only from real GitLab issue, job-log, and repository evidence, and label every guess as a hypothesis.
• Escalate on identity changes, delete or replace plans, contradictory evidence, or uncertainty.
• Open at most one focused merge request per incident.
• Act on a human's escalation decision only when it uses the exact wording decision: revert or decision: adopt.

One known gap: in the hackathon project the main branch is not protected against Developer pushes, so the flow's service account could technically commit to main. Nothing reaches the cloud without the human-triggered apply, but in a real deployment main would be protected.

• GitLab Duo Agent Platform custom flow with investigator and remediator agents
• GitLab project skill for the decision policy
• GitLab CI/CD schedules, issues, merge requests, manual approval gate, environments, resource groups, and Terraform reports
• GitLab-managed OpenTofu state
• Google Cloud Run
• Google Workload Identity Federation through GitLab OIDC tokens
• OpenTofu, shell, and jq

Accomplishments that we're proud of

We evaluated four controlled scenarios against the live Google Cloud deployment:

• Capacity increase: correctly adopted as low risk.
• Internal API authentication disabled: correctly reverted as high risk.
• No drift: created no issue and consumed no agent run.
• Internal service identity changed: correctly escalated until a human supplied an explicit decision.

All four scenarios correctly identified drift, or its absence. In every drift case the agent’s findings matched the pipeline evidence, and post-apply verification confirmed a clean plan. Every agent merge request passed validation and policy checks. The first one did so only after we fixed a CI permission issue that the run uncovered.

In the no-drift case, the check completed in 46 seconds without starting an agent session. In drift cases, the evidence issue was opened 40–58 seconds after the pipeline started, and an agent merge request was prepared in under two minutes after that.

For these tests we started the drift check by hand right after each change. On the real 6-hour schedule, detection itself can take up to 6 hours.

Challenges we ran into

Cloud-provider plans include noisy server bookkeeping and can expose sensitive values. We had to distinguish real, configuration-managed drift from provider read-back noise, redact sensitive data before it reached issues, and prevent personal information from appearing in incident evidence.

The first agent merge request failed CI: the flow’s service account can read GitLab-managed state but not lock it. Plans now run without a lock, while apply still locks.

We also found that a simple “revert high-risk drift” rule was insufficient: a runtime identity change can be dangerous to both adopt and revert. That led to the explicit escalation-first policy and human decision path.

What we learned

Reliable infrastructure automation needs an evidence boundary. The agent should use the issue, job log, and declared configuration; uncertainty and contradictory evidence should result in escalation.

We also learned that the security model must hold independently of the model. Short-lived cloud credentials, a separate read-only plan identity, a human-triggered apply job, and CI policy checks keep the workflow safe when an agent makes a poor recommendation.

What's next for Cloud Infrastructure Drift Detective

We plan to reduce the detection window with event-driven triggers alongside the six-hour schedule, expand the policy set beyond Cloud Run, and support more remediation patterns while retaining the same evidence and approval model.

In a production GitLab setup, we would protect main and the demo environment and require merge request approval before a production apply.

Built With

• and-terraform-reports-gitlab-managed-opentofu-state-google-cloud-run-google-workload-identity-federation-through-gitlab-oidc-tokens-opentofu
• ci/cd
• duoagentplatform
• environments
• gitlab
• google-cloud-run
• google-workload-identity-federation
• issues
• jq
• manual-approval-gate
• merge-requests
• mergerequests
• observability
• opentelemetry
• opentofu
• resource-groups
• shell