Elenchos for GitLab CI

Elenchos pushes a canary through a gate you own that checks for it, so the director finds the green run that should have gone red.

Comment

The problem

Once a year an engineering director signs that the security checks are in place. When an auditor asks him to prove it for one repository, what he has is a green pipeline and a screenshot. A green pipeline is a claim, not a control. A pipeline with a scanner job set to allow_failure: true, or a script ending in || true, can report green even when the scanner found something.

What it does

Elenchos tests the claim instead of trusting it. It plants a synthetic canary that a gate is supposed to catch, runs the pipeline, and reads what the gate actually did. The result lands where the director already works: a GitLab issue, a fix merge request, and a note on both after the fix is merged.

On GitLab it runs as a custom flow on the GitLab Duo Agent Platform (AI Catalog item 1018195: https://gitlab.com/explore/ai-catalog/flows/1018195/), started by a project trigger when a pipeline passes. The configured trigger is 3050395, with a successful-pipeline filter. Nobody asks it to run.

Most Creative: the counterfactual experiment

The useful synthesis is an experiment with three arms: the original gate, a repaired gate with the identical canary, and the repaired gate with clean input. This separates a masked failure from a repair that breaks every build. The recorded before, repaired and clean pipelines make the comparison inspectable. After the human merges the fix, the agent challenges the merged gate again.

Most Stages Covered: one loop, nine lifecycle stages

The workflow connects planning and creating a repair to verification, security, configuration, monitoring and governance: it files the issue, opens the fix MR, waits for the human merge and tests the merged gate again. The project pipeline also exercises packaging and release checks. The nine-stage map below links each claim to source and recorded evidence. Cloud Run deployment itself is manual. The concrete trail is Issue #2, MR !7 and the post-merge confirmation pipeline.

Recorded GitLab proof

The recorded run, all public:

  • The gate: job demo-neutered-scan exits 1 when it finds the planted secret, but ends in allow_failure: true (.gitlab-ci.yml at 4991f8e6).
  • 2026-10-14: the flow's service account filed issue #2.
  • 2026-10-15: after a passed pipeline (2950197588) the flow (Duo workflow pipeline 2950211507) ran three arms with the same canary blob on before and repaired:
    • before, pipeline 2950216951: scan job failed, allowed to fail, pipeline passed.
    • repaired, pipeline 2950219881: allow_failure removed, same canary, pipeline failed.
    • clean, pipeline 2950222473: repaired gate, no canary, pipeline passed.
    • The service account then opened MR !7 with a one-line fix.
  • 2026-10-16: I reviewed the outcome and merged MR !7 (merge commit a58762a0).
  • 2026-10-17: a passed pipeline on main after the merge (2958855350) started the flow again (2958861516). It planted the same canary on the merged gate; confirm pipeline 2958863173 failed as required, and the agent posted CONFIRMED_ENFORCING on issue #2 and MR !7.

Why it is supervised

You approve the outcome, not the steps. The agent plans, plants, runs three pipelines, decides the verdict, files the issue and writes the fix without asking. The flow has no merge step: the merge is the human decision, and after it the agent checks the fix again on main.

How I built it

  • Flow: .gitlab/duo/flows/elenchos.yaml, with agent steps for assessment, reporting and confirmation, and a deterministic step that runs the three-arm proof and routes on its one-word verdict.
  • Runner bridge: scripts/duo_loop.py reads the project and commit from the CI environment.
  • GitLab adapter: src/elenchos/forge/gitlab.py reads .gitlab-ci.yml, plants canary branches and reads job-level results.
  • CI: five stages (test, secure, package, gates, release), seven jobs, in .gitlab-ci.yml.
  • Google Cloud Run: src/elenchos/cloudrun.py, Dockerfile and deploy/gcp/cloudrun-service.yaml, deployed manually from the repository. /api/healthz, /version, /proof and /DEPLOYED_SHA report the recorded application build 7500a6994823cad2ee7c8de09d21bf2be571f0d5 (Google Cloud Run's Google Frontend reserves /healthz at the edge, so /api/healthz is the public health endpoint).

Public Google Cloud deployment

The public console is built from this repository's Dockerfile; the deployment manifest is public. Judges can inspect the console and its /DEPLOYED_SHA without an account. Deployment is manual, and the console does not push canaries. This is the Google Cloud deployment evidence for the optional bonus.

What I added during the Submission Period (Path B)

The original project, Elenchos, is MIT licensed and tested GitHub Actions workflows. Baseline: commit 7088f7c, the last commit before the GitLab import (dated 2026-10-11). The commits from 2026-10-08 to 2026-10-11 were general Elenchos work. At the baseline there was no .gitlab-ci.yml, no GitLab adapter, no Duo flow and no Google Cloud deployment.

At application build 7500a699: 88 commits since the baseline, the first on 2026-10-12. New files: .gitlab-ci.yml, .gitlab/duo/flows/elenchos.yaml and src/elenchos/forge/gitlab.py (2026-10-12); .gitlab/duo/agent-config.yml and scripts/duo_loop.py (2026-10-13); src/elenchos/cloudrun.py, Dockerfile and deploy/gcp/cloudrun-service.yaml (2026-10-18). Some existing files, such as the provisioning wrapper and the readiness script, were extended. All of the agent automation and the deployment are new work after 2026-10-05.

On 2026-10-27, MR !24 refreshed the README and presentation assets. This documentation update leaves the application and recorded video at build 7500a699; it does not represent a new runtime deployment.

Known limits

  • The configured pipeline-passed trigger is paused during the submission freeze; the public console and recorded proof remain available.
  • All non-merge authored Elenchos commits after baseline 7088f7c carry a DCO 1.1 Signed-off-by trailer; some merge commits, including a41b7141, do not.
  • GitLab lists my account as author and merger of MR !7. The project activity shows the flow's service account opened it on 2026-10-15 at 09:41 UTC, and its commit cd7b5229 is authored by Elenchos.
  • On 2026-10-24 I added sections to the descriptions of issue #2 and MR !7. Those sections are mine, not the agent's; the agent's text is in the description history.
  • The proof and the confirm ran on earlier versions of the flow. Catalog version 1.4.1 was published on 2026-10-18; since then it has fired on every passed pipeline on main and made no changes, because nothing was left to prove.
  • Duo workload logs are not public on GitLab, and GitLab's REST /notes endpoint returns 401 to unauthenticated callers. The evidence is what the agent produced: the pipelines, issue #2, MR !7, and the CONFIRMED_ENFORCING notes (#note_4035491242 on MR !7 and #note_4035491430 on issue #2) visible in the public GitLab web UI and in the demo video.
  • The public Cloud Run console reads; it never pushes a canary. Pushing happens only in the GitLab flow.

What's next

Run it across every repository in a group on a schedule, and hand the auditor the pipeline links as the evidence file.

LIFECYCLE STAGES (File:line references at 7500a699; historical pipeline/issue/MR IDs)

  • plan: .gitlab/duo/flows/elenchos.yaml:59 (create_issue), scripts/duo_loop.py:460, Issue #2 (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43277567/showcase/-/issues/2)
  • create: .gitlab/duo/flows/elenchos.yaml:60 (create_commit, create_merge_request), src/elenchos/prove/targets.py:74 (remove_mask), MR !7 commit cd7b5229 (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43277567/showcase/-/merge_requests/7)
  • verify: .gitlab-ci.yml:36 (stage: test), .gitlab-ci.yml:18-77 (demo-neutered-scan, agent-setup, unit-tests), src/elenchos/prove/experiment.py:560, pipelines 2950216951, 2950219881, 2950222473, 2958863173, 2986774918; unit-tests retains verification-report.json (no JUnit artifact)
  • secure: .gitlab-ci.yml:81 (stage: secure), .gitlab-ci.yml:18-45 (demo-neutered-scan), .gitlab-ci.yml:79-98 (secret-and-posture-scan), src/elenchos/provision/wrapper.py:266 (gitlab_security_report), pipelines 2950219881, 2958863173, 2986774918
  • configure: .gitlab-ci.yml:25-28 (removal of allow_failure: true), .gitlab/duo/agent-config.yml:7-22, deploy/gcp/cloudrun-service.yaml:1-53, MR !7 merge commit a58762a0
  • package: .gitlab-ci.yml:102 (stage: package), .gitlab-ci.yml:100-116 (package-wheel), pyproject.toml:1-37, Dockerfile:1-18, pipeline 2986774918; Dockerfile is built by manual Cloud Run source deployment; CI package-wheel smoke-tests HTTP handlers and retains cloudrun-smoke-receipt.json, without retaining a distribution wheel or SHA256SUMS.txt
  • release: .gitlab-ci.yml:139 (stage: release), .gitlab-ci.yml:137-153 (cloudrun-release-bundle), pipeline 2986774918; verifies the recorded bundle, checks public Cloud Run health, and retains cloudrun-release-manifest.json; deployment is manual
  • monitor: .gitlab/duo/flows/elenchos.yaml:100-115 (assessor -> prove -> confirmer), scripts/duo_loop.py:507 (confirm), configured trigger 3050395 (successful-pipeline filter); recorded main pipelines 2950197588 & 2958855350 and flow runs 2950211507, 2958861516, 2986781974 were started by the project's pipeline-passed trigger
  • govern: .gitlab-ci.yml:120 (stage: gates), .gitlab-ci.yml:118-135 (repository-gates), .gitlab/duo/flows/elenchos.yaml:63-70 (confirmer), src/elenchos/harness/governance.py:668, MR !7 (a58762a0), confirm arm 2958863173 (notes 4035491242 & 4035491430)

Built With

Share this project:

Updates

Submission history