RAS Falsifier

• 

  PASS - No regressions

• 

  Activity

Inspiration

A green CI pipeline does not necessarily mean that a code change is safe. Existing tests can pass while a small modification silently breaks behavior that was supposed to remain unchanged.

We built RAS Falsifier to challenge that assumption. Instead of simply asking whether a patch passes its existing tests, we ask: What did this patch change that it was not supposed to change?

What it does

RAS Falsifier is a supervised, evidence-driven regression detection workflow integrated with GitLab Duo.

It compares the original and modified versions of a program, explores additional inputs, and searches for unexpected behavior changes.

When it finds a potential regression, it attempts to reproduce it, checks whether execution reaches the modified code, and produces evidence for human review. For supported cases, it also prepares and checks a separate repair proposal.

If evidence is insufficient, it can abstain rather than claim certainty. It never approves or merges code automatically.

How we built it

We developed the public implementation from scratch in Python for Path A — Start Fresh / Supervised, without copying or depending on private RAS projects.

A custom GitLab Duo Flow is triggered when a merge request is created. The workflow analyzes the actual Git changes, runs the Falsifier engine, and posts a reviewable report back to GitLab.

GitLab CI/CD independently executes the project's tests, regression checks, verification procedures, and integrity controls.

Three different case studies

1. E-commerce checkout — real GitLab Duo demonstration

A developer lowers the free-shipping threshold for premium customers. All existing tests pass, but the change accidentally grants free shipping to standard customers.

RAS Falsifier explored 32 inputs, identified three intended behavior changes and one unexpected regression, and reproduced the defect. The changed-path evidence was verified, and a separate repair passed 10 acceptance checks.

View the real GitLab merge request

2. Document access control — local and CI demonstration

Editors should gain permission to export documents. Instead, an accidental change also allows ordinary viewers to delete documents.

Using the same Falsifier engine, the second example generated 64 role/action combinations and identified the unexpected permission change.

3. Order workflow — local and CI demonstration

Delayed orders should become eligible for retry. An incorrect modification also allows cancelled orders to be shipped.

The same unchanged engine explored 100 status/action combinations and reproduced the unauthorized transition.

The second and third examples are reproducible internal case studies, not additional live GitLab Duo merge-request sessions.

Challenges and what we learned

One challenge was making the workflow run inside GitLab Duo's actual environment, including resolving Python 3.9 compatibility issues.

We also strengthened the workflow to avoid reporting results from unrelated demonstration fixtures and revised our external testing protocol after discovering ambiguity in the original challenge labels.

The biggest lesson was that detecting an unexpected result is only the beginning. A useful tool must preserve the evidence, explain its limitations, and separate the proposed correction from its verification.

Accomplishments

• A real GitLab Duo-triggered workflow that analyzes a merge request and reports its findings.
• Three distinct Python case studies using the same falsification engine.
• Reproducible counterexamples even when historical tests remain green.
• Separate evidence verification and repair checks.
• A public MIT-licensed implementation with GitLab CI/CD integration.

What's next

We want to expand support for unfamiliar repositories, additional programming languages, richer test generation, and isolated application execution.

The current implementation remains a focused Python prototype, not a universal correctness verifier.

Green is not proof. Falsify before merge.

Source code: https://gitlab.com/ras-group4672841/ras-falsifier-lab

Built With

• ci/cd
• git
• gitlab
• python