ModelGate: supervised DevSecOps for ML models, run by GitLab Duo agents

Inspiration

In small teams, a model release is a person retraining in a notebook and copying a pickle to a server. Nobody checks whether accuracy dropped, whether the container ships the same model that was measured, or whether production data still looks like training data. "Life after code" for ML is mostly life after the notebook: the evaluation, security, release and monitoring work nobody has time for.

What it does

ModelGate makes every model change an MR that runs the whole DevSecOps lifecycle automatically, with GitLab Duo agents doing the toil and one human approving production (Supervised automation).

• Verify: ruff, 20 pytest tests, deterministic training, and an evaluation gate: the pipeline fails if accuracy or F1 drops more than 1 point vs. the approved baseline, falls below 0.90, or p95 latency exceeds 50 ms. The gate table is posted on the MR.
• Secure: SAST, Secret Detection, Dependency and Container Scanning.
• Package / Release / Deploy: the model is trained inside the Docker image (you ship what you measured), pushed to the GitLab registry, smoke-tested as a live service in staging, released on tags; production is when: manual.
• Monitor: /metrics for Prometheus and a PSI drift check per feature that opens a labelled drift issue.
• Govern: a release evidence card (metrics vs baseline, gate, drift, scanner findings, dependency licenses, final decision) published to GitLab Pages.

Four GitLab Duo Agent Platform custom flows close the loop:

1. MR Model Reviewer: runs the gate in its sandbox, looks for leakage, missing seeds, loosened thresholds and baseline edits, and posts one verdict.
2. Pipeline Fixer: classifies a failed pipeline, reproduces it, opens a fix MR, and is explicitly forbidden from "fixing" a model regression by loosening the gate.
3. Release Captain: gathers stage status, scans, drift and licenses, drafts release notes, then pauses at a HumanInputComponent approval gate before publishing anything.
4. Drift Responder: turns a PSI alert into a diagnosis (pipeline bug vs. real shift) and a fix or retraining MR that must pass the same gate.

Agents get facts, not logs: every job writes JSON evidence, and ModelGate ships its own MCP server (gate_policy, evaluate_candidate, read_evidence) wired in .gitlab/duo/mcp.json. AGENTS.md and chat-rules.md encode the guardrails every agent follows.

Proof it works (all public in the project)

• MR !1 (a one-line "harmless" regularisation change): the evaluation gate fails it (accuracy 0.986 → 0.867) and the MR Model Reviewer flow posts CHANGES-REQUESTED with the gate table.
• MR !2 → MR !3, no human in the loop: an API change broke lint and a unit test. The failed pipeline fired the Pipeline Fixer through a pipeline hook; it reproduced both failures, fixed the root cause (not the test, not the gate) and opened MR !3, which passed CI and was approved by Duo Code Review. (It fired once per failed pipeline, push and MR, so we closed the duplicate !4 and shipped Pipeline Fixer v1.1.0, which checks for an open fix MR on the same commit SHA before opening another.) Duo Code Review still asked for stronger tests on the feature MR, so instead of bypassing the review we added them (thresholds change predictions, out-of-range values return 422); Duo approved and MR !2 merged into main.
• MR !5, rejected on evidence: we tried a distroless runtime to cut container findings. CI now builds, scans and smoke-tests the image on any MR that touches it; every gate passed, but the scan went from 164 findings (44 High) to 302 (2 Critical, 61 High), so we closed it and kept python:3.11-slim.
• Drift: a simulated shift opens drift issue #2 with PSI per feature and hands it to the Drift Responder.
• Release Captain found two real bugs in our own v0.1.0 release (no scans on tag pipelines, a broken release description). We fixed them in v0.1.1; on v0.1.1 the Captain paused for approval, a human pressed Approve, and it posted the release notes on issue #3 while deploy_production stayed manual.

How we built it

Python 3.11, FastAPI, scikit-learn, GitLab CI/CD with security templates, Docker-in-Docker, GitLab Pages, GitLab Duo Agent Platform custom flows (flow registry v1), agent-config.yml, MCP.

Challenges

Designing agents that are useful but cannot cheat: the easiest way to turn a red ML pipeline green is to lower the bar. Every flow's prompt and AGENTS.md forbid touching the baseline or thresholds, and the baseline can only move through its own human-approved MR.

Accomplishments

• A single pipeline covering all nine DevSecOps stages for an ML service.
• A human approval gate built into an agent flow, not just a manual CI button.
• Evidence-first design: every decision is reproducible from artifacts.

What we learned

Agent autonomy is a policy question before it is a model question. Writing down what an agent may never do made the flows simpler and the review faster.

What's next

Per-slice fairness checks in the gate, shadow deployments that feed real traffic into the drift check, and a baseline-promotion flow that proposes a new baseline after N healthy production days.

Built with

gitlab-duo-agent-platform, gitlab-ci, mcp, python, fastapi, scikit-learn, docker, gitlab-pages

Built With

• docker
• fastapi
• gitlab-ci
• gitlab-duo
• gitlab-pages
• mcp
• python
• scikit-learn