Shakedown

• 

  Shakedown: a green build can lie. It refuses band-aids on every GitLab merge request by line, and proves real fixes on a real JVM.

• 

  A merge request adds retry: 2. Shakedown's gate refuses it in GitLab CI and names .gitlab-ci.yml line 37.

• 

  An @Ignore that passed 200 of 200 reruns, refused anyway at SentenceFactoryTest.java line 220.

• 

  The band-aid as a merge request in the public showcase project: one line, retry: 2.

• 

  The maintainers' real fix passes the gate; a human starts prove, which reruns it on a real JVM.

• 

  Our GitLab Duo Agent Platform flow file: read-only tools plus one MR note. Written, not yet enabled.

• 

  The pipeline deploys the dashboard to GitLab Pages with the recorded marine-api runs.

Inspiration

A flaky test passes and fails on the same code. The fastest way to turn a red build green is a band-aid: retry: 2 on the job, allow_failure: true, || true, an @Ignore, a sleep. Each one makes the build green and leaves the bug in place, and anything that only looks at the pipeline colour approves all of them. We built Flakeproof in September to catch that in Java tests. For this hackathon we turned it into automation that lives where the band-aids actually get merged: GitLab merge requests.

What it does

Shakedown runs on every merge request.

• shakedown-gate judges the merge request's own diff and every candidate fix it adds under candidates/. It refuses band-aids and names the exact file and line:
  • GitLab's own masks: retry:, allow_failure, || true, skipped tests
  • test masks: @Ignore, sleeps, retries

It also refuses changes to the gate's own CI job, so a merge request can't simply switch the check off. The pipeline goes red with the reason.

• prove reruns a clean candidate fix for a real open-source project, marine-api, under rotating test orders on a GitLab runner. It's manual, so a human starts it. Only a fix that passes counts as verified.
• The Shakedown GitLab Duo flow is a Duo Agent Platform custom flow, written to be requested as a reviewer. It reads the gate's verdict from the pipeline and posts one plain-language note: what was refused, the line, why it's a band-aid, and what a real fix would do instead. It never commits, approves or merges. A human decides. The flow file is in the repo; GitLab.com rate-limited our connection on submission morning, so we couldn't enable it in the UI before the deadline.
• pages publishes the Flakeproof dashboard, with our recorded runs, on every push to main.

Autonomy level: Supervised. Deterministic code decides the verdict, not a model, so the same merge request always gets the same answer. Duo explains it and proposes the real fix. A person approves.

Results (real runs on marine-api)

Flakeproof judged three candidate fixes for a real order-dependent flaky test, with 200 reruns each (our recorded runs, Sept 13, 2026):

The @Ignore patch passed every rerun, so a check based on reruns alone would have merged it. Shakedown refuses it and names the line. On GitLab, the same three candidates are merge requests in our showcase project, judged by shakedown-gate in CI.

How we built it

• Gate: Python 3.12. The deterministic gate and band-aid scanner from Flakeproof, extended for GitLab CI syntax and tamper checks.
• Pipeline: .gitlab-ci.yml runs unit tests (72), GitLab secret detection, the gate on every merge request, the manual rerun proof on a Maven and JDK 11 image, and the Pages deploy.
• GitLab Duo Agent Platform custom flow (flow registry v1): written with GitLab's own tools to read the merge request, its diffs, the failing jobs and their logs, then post a note.
• Dashboard: React, built and published by the pipeline.

Challenges we ran into

• Refusing masks without refusing real changes: making the gate refuse GitLab's own masks without refusing legitimate pipeline changes. Retries limited to runner failures stay allowed.
• A Linux-only failure: a patch that applied on Windows failed on Linux runners because of line endings. We found it by reproducing the Linux path, and now apply patches byte-exact.
• Keeping the verdict deterministic: keeping a model out of the verdict. Duo explains and proposes, but the decision is code.

Accomplishments that we're proud of

• On real code, Shakedown refused a patch that passed 200 out of 200 reruns, because it was an @Ignore. It named the line.
• A merge request that adds retry: 2 to make a flaky pipeline pass goes red, with the line named.

What we learned

A green pipeline is a claim, not proof. The check has to look at how the build went green, not just that it did.

What's next for Shakedown

• Let the Duo flow open its proposed fix as a new merge request once a human approves.
• Run prove automatically for small candidates.
• Handle more kinds of flaky tests beyond order-dependent ones.

What we added on or after Oct 5 (Path B)

.gitlab-ci.yml (unit tests, secret detection, shakedown-gate on every merge request, the manual prove job, the Pages deploy), ci/gate_mr.py with its JUnit report, the gate's GitLab CI band-aid and tamper checks, byte-exact patch apply for Linux runners, the Duo flow file, and the three demo merge requests. Flakeproof itself is from September.

Built With

• gitlab
• gitlab-ci
• gitlab-duo
• gitlab-pages
• java
• junit
• maven
• python
• react
• tailwindcss
• typescript
• vite