Pilot — Supervised AI Security Remediation

GitLab Duo proposes security fixes. Pilot’s deterministic G0–G7 gate verifies patches, blocks unsafe changes, and keeps final approval in human hands.

Comment

Pilot — Supervised AI Security Remediation for GitLab Duo

AI proposes. Deterministic verification decides. Humans retain final approval.

Inspiration

Generative AI coding agents are rapidly demonstrating proficiency in discovering vulnerabilities and suggesting remediation patches. However, in security-critical software engineering, generating a plausible patch is fundamentally distinct from proving that a system is secure.

Large Language Models (LLMs) optimize for plausible code completion, not deterministic security invariant preservation. A seemingly reasonable remediation proposed by an autonomous agent can easily:

  1. Introduce superficial evasions (such as quote-doubling sanitization) that satisfy simple test assertions while preserving the underlying dynamic SQL injection vector.
  2. Tamper with test suites, dependency locks, or CI configurations to force green pipeline runs.
  3. Mishandle exposed secrets by hardcoding replacement tokens.

Allowing an AI agent to self-certify or autonomously merge its own remediation introduces significant security and governance risks for enterprise teams.

This motivated our core architectural research question: How can engineering teams leverage the velocity of GitLab Duo's AI agents without granting an LLM final merge authority or self-approval rights?

That inquiry inspired Pilot: an architecture founded on strict separation of concerns:

$$ \text{AI Proposes} \implies \text{Deterministic Gate Decides} \implies \text{Humans Authorize} $$

What it does

Pilot establishes an automated, deterministic supervisory boundary around AI-generated code proposals. Rather than relying on model explanations, Pilot subjects candidates to an isolated, multi-layered deterministic gate sequence before presenting the change to a human reviewer.

[Red Team Scan] ──► Discovers Vulnerability
       │
       ▼
[GitLab Duo Agent] ──► Proposes Minimal Candidate Patch (C)
       │
       ▼
[Disposable Workspace] ──► Evaluates Candidate in Isolation
       │
       ▼
[Deterministic Gate] ──► Evaluates G0 through G7
       │
       ▼
[Evidence Digest] ──► Verifies Canonical SHA-256 Evidence Integrity
       │
       ▼
[Human Review] ──► Human-Authorized GitLab Merge

The 5-Stage Remediation Lifecycle

  1. Red — Scan: Identifies a security vulnerability in a target application.
  2. Blue — Propose: GitLab Duo Agent Platform (via custom flows or workflow sessions) proposes a candidate patch.
  3. Sandbox — Apply: The candidate is applied strictly within a disposable, isolated verification workspace—never written directly to production branches.
  4. Verification — Gate (G0–G7): The candidate is evaluated deterministically across eight rigorous verification layers.
  5. Release Decision — Review: The system generates a machine-readable verdict backed by a canonical SHA-256 evidence bundle. Final approval remains strictly with the human reviewer.

The Deterministic G0–G7 Gate Sequence

  • G0 — ScopeGuard: Enforces modification boundaries on candidate diffs. The candidate is restricted to application paths with a diff ceiling of 60 lines. Blocks tampering with .gitlab/, test suites, or CI configurations.
  • G1 — ASTSyntax: Confirms clean Python syntax trees and bytecode compilation across modified units.
  • G2 — ASTPolicy: Inspects Python syntax trees to reject unsafe dynamic SQL construction, including f-strings, non-constant string concatenation, .format(), and % formatting. It also checks parameter binding and imports against an approved allowlist.
  • G3 — BanditSAST: Executes pinned Bandit 1.8.6 static analysis in an isolated sandbox. If an autofixable issue (e.g., B608 SQL injection) remains unresolved, the gate returns FAIL; if a non-autofixable finding (e.g., B105 hardcoded password) is detected, it returns ESCALATE.
  • G4 — BehaviorSmoke & Attack Corpus: Probes candidate runtime behavior using an isolated SQLite harness against the active SQL-injection corpus and a wildcard-search leakage input. The repository also includes a separate, disjoint 10-case held-out SQL-injection corpus, validated independently against the vulnerable fixture. This separation allows additional adversarial cases to be tested without mixing them into the gate's active corpus.
  • G5 — PytestSuite: Runs the test suite via JUnit XML validation with defused XML parsing; zero failures, errors, or skipped tests are tolerated.
  • G6 — Determinism5x: Executes layers G0 through G5 across five consecutive iterations over identical commit inputs to verify verdict stability:

$$ \forall i, j \in {1, 2, 3, 4, 5}, \quad \text{Digest}(V_i) = \text{Digest}(V_j) $$

  • G7 — SHA256Evidence: Generates pilot-evidence.json, containing the verification context, commit metadata, and verdict, together with a SHA-256 integrity digest recalculated from the canonical JSON representation:

$$ H_{\text{evidence}} = \text{SHA-256}(\text{canonicalJSON}(\text{evidence payload})) $$

The evidence binds to commit identifiers, pipeline/job context, policy digests, and verification results. This provides cryptographic integrity verification; it is an unkeyed digest, not a digital signature.

Three Authoritative Verdicts

  • READY_FOR_HUMAN_REVIEW: The candidate passed all 8 deterministic checks and is eligible for human inspection. It does not trigger an automated merge.
  • BLOCKED: The candidate violated scope, failed AST policies, retained a vulnerability, or introduced a regression.
  • ESCALATE_TO_HUMAN: The finding (e.g., hardcoded credentials) cannot be safely automated and is escalated to human engineers with zero file edits committed.

How we built it

We designed and built Pilot from scratch during the hackathon to provide a complete, verified supervisor pattern for GitLab Duo:

  • Security Gate & Verification Engine: Developed in Python 3.11, leveraging the native ast module for syntax-tree inspection (astpolicy.py), pinned Bandit 1.8.6 for SAST scans, pytest with defusedxml for regression testing, and an isolated SQLite behavioral test harness.
  • GitLab Duo Custom Flows: Configured native GitLab Duo flow definitions (.gitlab/duo/flows/pilot-remediate.yml) with distinct BLUE_PROPOSAL and MR_AUDIT task modes, allowing Duo to act as the proposal engine while remaining completely decoupled from the decision boundary. For offline development and benchmarking, an optional Vertex AI backend (google-genai SDK) is supported under explicit configurations.
  • Frontend & Interfaces: Built a modern, reactive interface using React 18, TypeScript, Vite, and TailwindCSS for the primary Remediation Lab, alongside a complementary Flask-based Wargame Dashboard for live adversarial simulations.
  • Cloud Run Deployment & Identity: Deployed the application container to Google Cloud Run (pilot-dashboard). GitLab CI/CD authenticates to Google Cloud through GitLab OIDC and Workload Identity Federation (WIF), while the running Cloud Run service executes under a dedicated runtime service account.

The public demonstration on Cloud Run enforces strict boundaries: anonymous public requests are served from pre-seeded demonstration data and cannot trigger unauthorized external model invocations.

Challenges we ran into

  1. Detecting Plausible Evasions (The Quote-Doubling Trap):
    AI models frequently propose email.replace("'", "''") to fix SQL injection. Standard test suites pass because sanitized queries execute without syntax errors. Building G2 (AST inspection) was necessary to parse the syntax tree and reject string interpolation regardless of test outcomes.
  2. Preventing Agent Self-Certification & Scope Drift:
    When an autonomous agent encounters a failing regression test, its default instinct is often to modify the test assertion or alter CI configuration. Enforcing G0 ensures that any patch touching files outside the authorized application boundary (app/) is rejected immediately.
  3. Achieving Deterministic Repeatability Across Non-Deterministic AI:
    LLM output is stochastic. We resolved this tension by decoupling proposal generation from verification: while proposal generation uses AI, the verification gate (G0–G7) is strictly deterministic, offline, and zero-network.

Accomplishments We're Proud Of

Verified GitLab CI/CD Integration (MR !15)
A fix addressing application-file preservation during Wargame patching was validated by its merge-request pipeline and merged into main.

Deterministic Rejection of Unsafe Remediation (MR !8)
An unsafe quote-doubling workaround was blocked by the deterministic AST policy and security checks, demonstrating why passing ordinary tests alone is insufficient.

Human Escalation for Sensitive Findings (MR !9)
A hardcoded-credential finding triggered human escalation rather than an automatic security approval, preserving human authority over sensitive decisions.

Supervised SQL Remediation (MR !7)
A parameterized SQL remediation was evaluated through the verification workflow. The candidate's readiness verdict remains distinct from final merge approval.

Repeatable Local Verification
Pilot's local verification workflow is designed to evaluate proposed changes through deterministic checks, with repeatability and evidence integrity as explicit goals.

Live Public Demonstration
The public Cloud Run deployment demonstrates the Remediation Lab and Wargame interface. The accompanying demo video shows the workflow and its recorded outcomes.

What we learned

  • AI proposes; deterministic code decides. Generative models are creative proposal engines, but they cannot act as compliance authorities for their own work.
  • Negative verdicts represent architectural success. A security workflow that rejects an unsafe quote-doubling patch or escalates an exposed credential is demonstrably succeeding at its objective.
  • Enterprise adoption demands mathematical trust boundaries. Regulated industries (financial technology, healthcare, critical infrastructure) cannot adopt autonomous coding agents without auditable, deterministic gates that guarantee human authorization before code reaches production branches.

What's next for Pilot

  • Multi-Language AST Support: Extend the G2 policy engine beyond Python to support Go, Java, and TypeScript AST parsing.
  • Micro-VM Sandboxing: Integrate lightweight virtualization (Firecracker / gVisor) for deeper behavioral isolation of untrusted candidate runtime execution.
  • Duo Workflow Catalog Integration: Package Pilot as a standard, reusable compliance gate component in the official GitLab Duo Flow registry.

Pilot does not assume AI will always be right. It is engineered to verify what AI proposes and stop when the evidence is not good enough.

Built With

Share this project:

Updates

Submission history