FlakeTrail

Turn an order-dependent test failure into a smaller reproducer and check a repair that preserves every assertion.

Comment

FlakeTrail — From Failing CI to a Verified Repair

Inspiration

A test suite can pass consistently when tests run individually yet fail when they execute together. These order-dependent failures are particularly frustrating because the problem may not be inside the failing test itself.

One test changes shared state. Another test inherits that state. The resulting failure can be difficult to reproduce, diagnose, and fix without losing valuable test coverage.

The tempting shortcut is to skip the troublesome test, add retries, or rearrange execution until CI turns green. But a green pipeline does not necessarily mean the underlying problem has been fixed.

I built FlakeTrail to explore a better approach: reproduce the failure, reduce it to the smallest demonstrated interaction, identify the state contamination, and verify a repair without deleting assertions.

The project connects that investigation to a real GitLab Duo Agent Platform and GitLab CI/CD workflow.

What it does

FlakeTrail is a browser-based test-failure investigation and repair-verification workbench.

It helps developers understand why a test fails only after another test has run.

Its workflow includes:

  • Failure reproduction: Replay the recorded test order in a fresh Python subprocess.
  • Isolation checks: Confirm that the nominated failing test passes when executed independently.
  • Failure minimization: Remove irrelevant tests while preserving the original failure fingerprint.
  • Reduction evidence: Record each attempted deletion, retained interaction, execution budget, and measured runtime.
  • Repair verification: Check an isolation fix against the reduced order and independently seeded full orders.
  • Coverage preservation: Reject repairs that merely skip the failing test or remove required assertions.
  • Transparent outcomes: Return an unresolved result when the execution budget is exhausted rather than claiming a successful diagnosis.

The application executes only predefined, repository-owned test fixtures. It does not accept arbitrary source execution or shell commands.

Reports can be exported as JSON or explicitly saved, restored, and deleted within the current browser.

How I built it

1. Starting with a real failing GitLab CI job

I used an original Python checkout fixture containing an intentionally planted shared-state problem.

The original GitLab CI execution captured this order:

health → set_eur → inventory → default_checkout → receipt

The set_eur test changed currency-related state. Later, default_checkout failed because it expected the default currency to remain USD.

The real GitLab job recorded five assertions: four passed and one failed.

This was a deliberately introduced fault in an owned example, not an undiscovered production incident.

2. Building the failure-reduction engine

The analysis engine uses Python 3.14, standard-library tools, and bounded subprocess execution.

It checks the original failure, verifies independent execution, and attempts to remove unnecessary tests without losing the observed failure.

The verified candidate investigation performed 13 minimization probes, retaining the two-test interaction:

set_eur → default_checkout

Every individual deletion from the remaining sequence was checked. This establishes deletion-minimality for the measured scenario, not a mathematically guaranteed globally shortest reproducer.

3. Implementing a repair without weakening coverage

The owned example includes an isolation repair that restores shared state and the fixture environment after each test.

Instead of hiding the failing test, the verification process checks that the original assertions continue to execute.

The candidate repair passed 11 orders, including ten independently seeded full orders.

Two negative controls, including one that skipped the failing test, were rejected.

4. Integrating GitLab Duo and CI/CD

FlakeTrail also completed a real GitLab sponsor workflow.

A successful GitLab Duo Agent Platform session analyzed the captured failure evidence and supported diagnosis and review.

The reviewed candidate was promoted to a dedicated branch and draft merge request.

The proposed change affected only review-fixture/isolation.json. The original assertions and failure-capture evidence remained intact.

A genuine GitLab CI candidate pipeline then verified the repair.

The candidate pipeline included:

  • 14/14 passing repair assertions
  • 13 minimization probes
  • 11 accepted repair orders
  • Two rejected negative controls
  • A deliberately retained failing capture marked allowFailure=true

The blocking repair job passed while the original failing example remained visible.

The draft merge request remains unmerged, preserving human control over final acceptance.

5. Deploying the application

I deployed the public workbench using Docker, Google Cloud Build, Artifact Registry, and Google Cloud Run.

The frontend uses JavaScript, HTML, and CSS.

The stateless server does not maintain a database of user reports. Optional report persistence happens within the user's browser.

The application also includes bounded request and subprocess execution, browser-origin restrictions, and a restrictive content security policy.

Live application: https://flaketrail-kc4mxxrtbq-uc.a.run.app

Challenges I faced

Reproducing the actual failure

A failing test is not necessarily the source of its own failure. The first challenge was reproducing the precise interaction from CI rather than making assumptions based on the error message.

Reducing the sequence without changing the problem

Removing tests too aggressively can eliminate the failure entirely. The reduction process needed to preserve the original failure fingerprint and independently verify attempted deletions.

Avoiding false-positive repairs

A pipeline can turn green simply because the problematic test was skipped. I built explicit negative controls to distinguish a genuine isolation repair from lost test coverage.

Preserving truthful CI evidence

The candidate pipeline deliberately retains the original failing capture while separately verifying the repaired behavior.

That distinction matters: an expected failing diagnostic job should not be misrepresented as an unsuccessful repair, nor should it be hidden to make the pipeline appear cleaner.

Connecting AI analysis with real provider execution

A simulated agent interaction is not equivalent to a real GitLab Duo session.

The workflow required genuine provider execution, traceable merge-request evidence, authenticated CI artifacts, and clear boundaries around what the agent actually contributed.

Accomplishments

FlakeTrail reached a publicly deployed and technically verified release.

Key results include:

  • Genuine failing GitLab CI evidence
  • Successful GitLab Duo agent session
  • Real draft merge request containing the reviewed candidate
  • Successful candidate CI pipeline
  • Two-test deletion-minimal reproducer in the controlled fixture
  • 14/14 passing candidate repair assertions
  • Ten independently seeded full-order repair checks
  • Negative controls that reject lost test coverage
  • Nine successful public browser workflows and six HTTP/security checks

The final public presentation was also verified, including its investigation results, repair evidence, and actual GitLab workflow links.

What I learned

The key engineering lesson is that making CI green and proving a repair are different problems.

A reliable investigation needs a reproducible failure, an inspectable reduction trail, preserved assertions, meaningful negative controls, and evidence that survives beyond a single successful run.

AI-assisted DevSecOps becomes more useful when agent findings are connected to actual jobs, reproducible artifacts, and human-reviewed changes rather than accepted without verification.

What's next

Future improvements could include supporting additional repository-owned test frameworks, improving failure-fingerprint analysis, and evaluating more complicated shared-state interactions.

A broader benchmark is also needed to measure reduction effectiveness and repair reliability beyond the original controlled fixtures.

FlakeTrail does not claim that finite testing eliminates every possible flaky failure. Its goal is to make investigation and repair evidence more reproducible, reviewable, and trustworthy.

Try FlakeTrail

Live application: https://flaketrail-kc4mxxrtbq-uc.a.run.app

GitLab repository: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43478475/flaketrail

GitLab Duo execution: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43478475/flaketrail/-/automate/agent-sessions/10031632

Verified candidate pipeline: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43478475/flaketrail/-/pipelines/2988163158

FlakeTrail demonstrates a real AI-assisted GitLab workflow, from a captured failing CI job to a minimized reproducer and a separately verified, reviewable repair.

Built With

Share this project:

Updates

Submission history