Flake Hunter

FlakeHunter is a GitLab Duo flow that works out, from re-run evidence, whether a red pipeline is a real bug, a flaky test or an infrastructure blip.

Comment

Inspiration

Every red pipeline asks the same question: is it my change, the test, or the runner? A flaky test passes and fails on the same code, so teams re-run and hope, or quarantine without limits and quietly lose coverage. Flaky-test detection is still an open proposal in GitLab itself (#381799). We wanted a flow that answers the question from evidence, finishes the job, and leaves a human in charge of the outcome.

What it does

When a pipeline fails, a CI job re-runs each failing test 10x alone and 10x in suite order, and a plain, tested script decides the verdict. The model never classifies a failure. The FlakeHunter flow, running in GitLab Duo Agent Platform, then acts on that verdict:

  • FLAKY: opens a quarantine branch, MR and evidence issue, writes a root-cause fix, proves it with 20 consecutive passes, and asks a human to Approve / Modify / Reject. On approval it removes the quarantine marker. A human merges.
  • REGRESSION: hands off with the evidence. Nothing is quarantined.
  • INFRA: explains the infrastructure signature and suggests a retry:when snippet; no code change.
  • OVER_BUDGET: refuses to quarantine when the budget is full.

Containment is governed: a budget (5 tests or 2% of the suite, whichever is larger), a 14-day expiry that fails the pipeline if a quarantine is forgotten, and a non-blocking job that keeps running quarantined tests. After a merge, CI creates a GitLab Release and records the recovery in the Flake Ledger. Autonomy level: Supervised (you approve the outcome, not the steps).

How we used GitLab across the lifecycle

Plan: evidence issues with owner and expiry. Create: agent fix commits. Verify: the re-run probe and the 20-pass proof. Secure: SAST and Secret Detection on the agent's own branches. Package: a Cloud Build image. Release: a GitLab Release per merged fix. Configure: quarantine markers and retry:when as code. Monitor: a live dashboard with mean time to recovery. Govern: a per-step tool allow-list, a budget block, expiry enforcement and audit notes. Links and screenshots for every stage are in docs/evidence.md in the repository.

Built with (what each tool does here)

  • GitLab Duo Agent Platform: the custom flow (collect, verdict, contain, fix, prove, human gate, report), the pipeline-failed trigger, a service account, the human-approval step, and the sandbox config.
  • GitLab CI/CD: runs the probe, the quarantine and expiry jobs, deploys, creates releases and syncs the Ledger, with SAST and Secret Detection for the Secure stage.
  • Python and pytest: the demo app and tests, plus the probe, classifier, release and carbon scripts (standard library only).
  • FastAPI: the Flake Ledger's API and dashboard.
  • Firestore: serverless storage for the Ledger (the container filesystem is ephemeral, so our first SQLite version lost data).
  • Google Cloud Run: hosts the Ledger and scales to zero.
  • Cloud Build and Artifact Registry: build and store the image on demand.
  • Workload Identity Federation: GitLab's OIDC token is exchanged for short-lived Google credentials, so there are no stored keys.

Challenges we ran into

  • The flow's tools are GitLab-only. It cannot call an outside URL, so anything that needs the Ledger or a computed verdict runs in CI and the flow reads the result.
  • A constrained sandbox. The flow's image has Python 3.9 and no pip, so our first proof ran no tests. We now build a pytest virtual environment in the sandbox setup.
  • A strict flow schema. The registry rejected a misplaced default_route, and routing needs an exact token, so classification had to be a single, bare output.
  • A reproducible demo. A realistic flake fails intermittently: often enough to turn the pipeline red on demand, but never always (that would be a regression).
  • Platform behaviour we had to discover. Branch pipelines carry no merge-request number, protected CI variables vanish on unprotected branches, and anonymous API calls are rate-limited.

What testing taught us

Running the real loop found bugs that unit tests never would. Every one is fixed and listed in the repo's evidence log:

  • A guard that cannot run must say so. The budget check was silently skipped on unprotected branches, so a full budget was ignored. It now records whether the check actually ran.
  • Fail loudly, not silently. Two flow steps ended with no output when they could not find the MR. They now look it up by branch and fall back to an issue.
  • Anything that creates a git ref triggers CI. Creating release tags ran pipelines on old commits and started the flow on stale code. Tag pipelines are now excluded.
  • Event-driven agents need idempotence. The flow would have opened duplicate quarantine MRs for a test that already had one.
  • Honest proof beats confident proof. The first proof came back 0/20 because the sandbox could not run tests. The agent said so, a human sent it back with Modify, and the second proof was 20/20.
  • Check your own claims. We dropped a statistic we could not verify, and made the demo dry-runs genuinely inert after one wrote files into the working tree.

What's next

  • Beyond pytest. The re-run idea is language-agnostic, but our probe is not. We would make the test command and result parser configurable (JUnit XML first), so any CI suite can use the same classifier and flow.
  • Cheaper evidence. Replace the fixed 10 + 10 re-runs with sequential sampling that stops as soon as the verdict is clear. The probe is the largest part of FlakeHunter's own CI time, so this lowers both cost and carbon.
  • A flake history in the Ledger. Per-test trends, first-seen dates, automatic detection when a fixed test flakes again, and one budget shared across repositories.
  • Better signal in. Use GitLab's native flaky-test detection (#381799) as an input once it ships, and use test-to-code ownership to route an evidence issue to the right owner instead of the last committer.
  • The Ledger as an MCP server, so other Duo agents can query quarantine status and history.
  • Policy-driven approval. Keep Supervised as the default, but let teams pre-approve low-risk fixes (for example 20/20 plus a passing suite), while still never merging automatically.

Try it

Public GitLab project: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43285461/flake-hunter Live Flake Ledger (Google Cloud Run): https://flake-ledger-c2fxyebmkq-uc.a.run.app

Built With

  • artifact-registry
  • cloud-build
  • fastapi
  • firestore
  • gitlab-ci-cd
  • gitlab-duo-agent-platform
  • google-cloud-run
  • pytest
  • python
  • sast
  • secret-detection
  • workload-identity-federation
Share this project:

Updates

Submission history