Inspiration
I spent months testing my own trading ideas against market data. Every idea that looked good fell apart once I stopped adjusting it after seeing the results. Nothing in my tools stopped me from changing a filter after seeing a number, so I kept fooling myself. Scientists call the fix pre-registration. I wanted pre-registration that a pipeline enforces, instead of a promise.
What it does
A question goes into a GitLab issue. A custom Duo flow with three agents turns it into a study: an author agent writes the plan, the analysis code and its tests without access to the data (the agent runtime's network policy blocks the data host), a read-only auditor agent lists p-hacking risks, and a publisher agent fixes them and opens a draft merge request. The merge request pipeline checks the plan's rules, simulates false-positive rate and power, runs SAST and secret detection, and waits for GitLab Duo Code Review, which applies statistics-specific rules kept in the repository. On merge, CI seals the plan and code as a GitLab Release, then fetches the data, runs the sealed code once and records the verdict as a second Release. The verdict is posted on the issue, a second flow writes a plain-language report, and CI rejects wording the verdict does not support. A daily schedule re-hashes every seal and re-downloads every data source, and opens an issue if anything drifted. A registry of every seal, verdict and prior look is published with GitLab Pages.
This all happened for real in the showcase project: issue #2 asked whether Chinstrap penguins have longer flippers than Adelie penguins. The flow wrote study 0004 and opened !1. Duo Code Review asked for two changes (pin the data URL to a commit, state the test and the required sample size plainly), re-reviewed the fix and approved. The merge sealed the study (Release seal-0004), the run recorded "evidence of an effect", a 5.87 mm difference with p = 7.5e-9 (Release verdict-0004), the reporter flow wrote the report in !3, and merging it closed the issue. Merge request !2 shows the other side: an outlier filter added to a sealed analysis fails the seal check and cannot be merged.
How we built it
Python engine (python -m sealed) with hashing, statistics (a Welch-style normal-approximation test or a permutation test, bootstrap interval, sample size), plan rules, an overclaim checker, a static registry and GitLab API calls that use only the CI job token. GitLab pieces: two custom Duo flows (flow registry v1, three agents and one agent), an agent runtime configuration with a network deny list, AGENTS.md, two Duo skills, Duo Code Review instructions, CI/CD with SAST and secret detection, Releases, Pages, the PyPI package registry and a pipeline schedule.
DevSecOps stages covered
Plan (issue template, the flow reads the issue), Create (author flow writes code and opens the merge request), Verify (unit tests, plan rules, simulation, claims check, Duo Code Review gate), Package (the engine is published to the project's PyPI registry), Secure (SAST, secret detection, review rules, the agent network deny list that keeps the author blind), Release (seal and verdict Releases), Configure (agent runtime config, CI variables, schedule), Monitor (scheduled drift check that opens issues), Govern (seal enforcement in the pipeline, review gate, prior-look disclosure in every verdict).
Challenges we ran into
Making "sealed" mean something: the hash covers the parsed plan and the exact bytes of the code, so reformatting the plan does not matter but any change to the code does. A study edited after its seal is reported as "plan violated" before any data is read. Keeping the author agent honest was the second challenge: the flow runs in the sandboxed default image, where the network policy denies the data host, so the agent cannot peek even if it wanted to. And GitLab's own rules caught real gaps: Duo Code Review pointed out that the first plan's data URL pointed at a moving branch, which is exactly the kind of hole a pre-registration must not have.
Accomplishments that we're proud of
The whole loop ran on GitLab without a human writing a line of the study: issue, three-agent flow, pipeline, Duo review and re-review, seal, single run, verdict, report, closed issue. The seed studies produce one of each verdict on the Palmer penguins data, and the engine's tests include a p-hacked analysis and an underpowered plan that the simulation check catches.
What we learned
Agents are good at the paperwork of rigor, as long as the pipeline, not the agent, decides what counts. A reviewer with written rules (Duo Code Review with custom instructions) is more useful than a reviewer with opinions.
What's next for Sealed
More test types, and mirroring seals to a public timestamp service so even the project owner cannot rewrite them.
Built with
GitLab Duo Agent Platform (custom flows, Code Review Flow), GitLab CI/CD, GitLab Releases, GitLab Pages, GitLab package registry, Python, NumPy, PyYAML.
Honesty note
The three seed studies (0001 to 0003) are demonstrations: their comparisons were computed before the plans were written, and each says so in its prior-look field. Study 0004 was written blind by the flow; its issue author had seen summary plots of the data in tutorials, which the plan and the verdict both disclose.
Log in or sign up for Devpost to join the conversation.