ModelGate: a model promotion gate for GitLab

Every model gets a merge request: CI judges a retrained model with published rules, a GitLab Duo flow writes its model card and opens the MR, and a person merges to promote it.

Comment

Inspiration

Small ML teams that already use GitLab review every line of application code, then ship a retrained model because one number went up. Nobody checks whether a customer segment got worse, the model card is written afterwards if at all, and nobody knows what the training run cost in carbon. Code already has a review path on GitLab. We wanted the same for models: every model gets a merge request.

What it does

ModelGate treats a retrained machine learning (ML) model like code. The example in the repository is a credit risk model that predicts how likely a customer is to miss loan payments in the next 12 months, trained on synthetic data.

  1. A person pushes a training change. GitLab CI trains a challenger and evaluates it on a fixed holdout, overall and for each segment of a protected attribute. It scans the model file (picklescan, with a module allowlist) and the dependency lock (pip-audit, with OSV severities), and measures the run's carbon cost as a Software Carbon Intensity figure.
  2. Six published rules give a deterministic GO, NEEDS REVIEW or NO-GO, and a version bump.
  3. The passed pipeline triggers a GitLab Duo custom flow. The agent works out which pipeline started it, stops when there is nothing to package, has code render the model card and decision record, writes the analysis and a three-line summary, fixes them when the commit step rejects them, and opens or updates the merge request, titled and labelled by the verdict. On GO and NEEDS REVIEW it requests a review.
  4. A second flow, ModelGate Ask, answers reviewers' questions on the merge request, such as "what would make this GO?", from the committed decision record.
  5. A person merges. CI confirms that main reproduces the approved metrics exactly, registers the version, creates a Release with the model card, records a production deployment and republishes the Pages dashboard. The flow posts "Promotion complete" on the merge request.
  6. A nightly schedule scores a drifted batch with the champion, and the dashboard raises an alert when a feature's PSI passes the limit.

Autonomy level: supervised. The agents run every step from the push to the merge request, and from the merge to the Release. A person approves one thing: the merge. ModelGate covers 8 of the 9 lifecycle stages: Create, Verify, Package, Secure, Release, Configure, Monitor and Govern.

See it running:

How we built it

  • GitLab CI/CD on GitLab-hosted runners. Every API call uses CI_JOB_TOKEN, so there are no CI/CD variables and no personal tokens.
  • GitLab Duo Agent Platform: two custom flows, ModelGate on a Pipeline events trigger and ModelGate Ask on a Mention trigger, each with its own service account. AGENTS.md holds their rules and .gitlab/duo/agent-config.yml their runtime.
  • A Python helper, python -m modelgate.flow, does every file and git operation for the agent. The agent writes only two blocks of text. The commit step re-renders the card and rejects any other change, any number that is not in the card or the decision record, and text that calls a failed rule passed.
  • Releases, Environments, Pages, pipeline schedules, the package registry, SAST and Secret Detection.
  • Python with scikit-learn, CodeCarbon, picklescan and pip-audit, on synthetic, seeded data.
  • Carbon follows the SCI specification (ISO/IEC 21031:2024), per evaluated candidate: SCI = (E x I + M) / R, with R = one candidate.

Challenges we ran into

  • Keeping the agent honest. Code renders every table and figure in the card, and the commit step refuses anything the agent changes outside its two blocks, any number it invents, and sentences that call a failed rule passed.
  • GitLab's flow form treats HTML comments in a prompt as dangerous content, and it rejected max_cycles, a field in GitLab's own v1 specification. The prompt now names its markers in words, and a test fails on any pattern the form rejects.
  • On a branch, CI's variables made the gate inside the unit tests read main's champion through the API, so 9 tests failed only on GitLab. The tests now run without CI's variables.
  • joblib model files cannot be fully scanned: picklescan stops at joblib's inline numpy arrays. ModelGate saves a standard pickle that it can parse end to end.
  • A branch that earns a GO and is then pushed again must not compare against itself. The gate reads the champion from the default branch, and the flow resets the branch's champion when a later run is not GO.

Accomplishments that we're proud of

  • The whole loop ran on GitLab with no person between the push and the merge request: GO, NO-GO and NEEDS REVIEW, each card and description written by the flow and accepted by the commit step's checks.
  • The NEEDS REVIEW candidate raised overall AUC from 0.8397 to 0.8558 while segment C fell from 0.8467 to 0.7107. A gate on the headline metric would have shipped it.
  • On !5 the same merge request went from NEEDS REVIEW to GO after one more training change. Merging it promoted 0.2.0: reproduced on main, released (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43712399/showcase/-/releases/model-v0.2.0) and deployed.
  • ModelGate Ask answered "what would make it GO?" on !4 with numbers that all match the decision record.
  • The gate caught real advisories in our own dependency lock: a CRITICAL one (certifi) before the first pipeline ran, and later a HIGH one (filelock) published after the lock was built. The same code, retrained unchanged, went from GO to NO-GO until the lock moved.
  • Carbon is reported for every candidate: 0.036 gCO2e median, 0.074 gCO2e to reach 0.2.0, and a docs-only push trains nothing.

What we learned

  • A Pipeline events trigger hands the flow the whole pipeline webhook payload as its goal.
  • The flow's token can download job artifacts, so the flow packages CI's own reports instead of retraining.
  • The Model Registry does not accept CI_JOB_TOKEN; the generic package registry does, and the job token can create Release tags.
  • Enabling a flow adds Mention and Assign triggers by default. Remove the ones the flow does not expect, or every mention starts a run.
  • Deleting a merged branch breaks links to the model card on it, so ModelGate's branches stay.

What's next for ModelGate: a model promotion gate for GitLab

  • Real datasets with their own fairness slices.
  • Comparing candidates against the champion on the champion's stored predictions, instead of a retrain.
  • The Plan stage: open an issue on NO-GO that says what to change.
  • Measuring the flow's own energy once GitLab exposes it.

Built With

  • agent
  • codecarbon
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • gitlab-pages
  • picklescan
  • pip-audit
  • python
  • scikit-learn
Share this project:

Updates

Submission history