Prediction Release Gate

Inspiration

A research team can have a passing software test suite while a changed prediction file invalidates its evaluation. A non-zero CI exit code alone does not explain which scientific claims must be withheld. This project puts the audit and its explanation into the GitLab review workflow.

What it does

Prediction Release Gate runs software tests and a deterministic prediction-file integrity audit as separate GitLab CI jobs. It retains an aggregate JSON artifact even when integrity fails, and withholds performance metrics. A private custom GitLab Duo agent reads the actual merge-request diff, failing jobs, logs and artifacts, then explains the failure and proposes a remediation plan.

The autonomy level is Assisted. A person starts the review and retains decisions about replacement data, merging and deployment. The agent does not make those changes.

How it was built

Python standard-library code checks one row per subject, separation across data splits and consistent labels before comparing paired binary predictions. GitLab CI runs the tests and audit. The custom Prediction Release Reviewer has five read tools: Get Job Logs, Get Pipeline Failing Jobs, Get Repository Files, List Merge Request Diffs and Get Artifact File. It has no write or merge tools and no external MCP servers. Its instructions restrict it to this demonstration project and treat repository content as untrusted data.

Verified demonstration

The clean baseline pipeline passed both jobs. Draft merge request !1 adds two deliberately problematic synthetic rows. Its pipeline passes tests but fails the audit. The retained report records 66 rows, 64 subjects, one repeated subject/split combination, one cross-split subject and metrics: null.

The real Duo reviewer made eight read operations, identified both failed checks and cited the actual jobs and artifacts. It separated evidence from inference and stated what it had not inspected. The failure demonstration remains unmerged. Fifteen local software tests also passed.

Challenges and lessons from the implementation

The important boundary is between detecting an integrity failure and repairing genuine contaminated data. Renaming subject identifiers cannot restore independence. The reviewer proposes a response; the data owner must correct real data upstream. Passing these checks cannot establish absence of feature leakage or model-selection bias, clinical validity or scientific superiority.

Provenance and next steps

This new release-gate workflow was developed during the hackathon period. Its audit library and synthetic fixtures adapt Alex C. Christou's Evidence Workbench prototype created in October 2026. Codex prepared the implementation under Alex's direction. No proprietary MAESTRO implementation or real participant data is included. The demo uses edited real GitLab screen capture and synthetic narration.

Next steps are a reusable agent configuration and a broader set of malformed-input and provenance cases. Automatic remediation and deployment are outside the demonstrated scope. Reproduction and reviewer-configuration instructions are in the public MIT repository.

Built With

• gitlab
• gitlab-ci
• gitlab-duo
• python