• 

  PolicyProof: Evidence Before Enforcement

• 

  Pipeline

• 

  Jobs

• 

  Tests

PolicyProof: Evidence Before Enforcement

Autonomy Track: Assisted Autonomy

Policy changes are software changes. They should be tested before they are trusted.

PolicyProof is a deterministic governance policy backtesting engine designed to help software teams evaluate policy changes against historical merge request examples before deciding whether to promote them. It brings a testing mindset to governance itself: compare decisions, examine potential regressions, measure classification behavior, and keep consequential decisions accountable to human reviewers.

Inspiration

Modern software delivery relies on automated checks, security requirements, merge request rules, and release controls. These mechanisms are essential, but the policies governing them can evolve independently of the application code they protect.

A seemingly small policy change can have consequences in both directions. A legitimate change might be blocked unnecessarily, creating friction for developers, while a risky change might escape a rule that was expected to catch it. The challenge is not merely writing a policy. It is understanding how that policy behaves across different cases before trusting it to govern future decisions.

This led us to a fundamental question:

What if governance policies could be tested like application code before they were promoted into enforcement?

That question became the foundation of PolicyProof.

We wanted to move policy evaluation away from intuition and isolated configuration edits toward repeatable experiments, baseline comparisons, measurable outcomes, and reviewable evidence. Rather than treating a policy change as a harmless configuration update, PolicyProof treats it as a behavioral change whose consequences deserve to be examined.

Our goal is to make governance more testable, transparent, and accountable without removing the human judgment that important software delivery decisions require.

What it does

PolicyProof evaluates structured governance policies against historical merge request examples and helps users understand how proposed rules affect policy decisions.

Its core capabilities include:

• Structured policy configuration: Define governance rules in YAML, making policy conditions explicit and easier to version, inspect, and maintain.
• Deterministic evaluation: Apply policy conditions consistently to a given set of inputs, supporting reproducible evaluation and debugging.
• Historical backtesting: Evaluate policy behavior against a collection of merge request examples before relying on a proposed policy.
• Baseline-versus-proposed comparison: Examine how a proposed policy changes decisions relative to an existing baseline.
• Regression analysis: Identify unintended changes in policy outcomes that warrant investigation before promotion.
• Evaluation metrics: Generate precision, recall, F1 score, and confusion-matrix information to help assess classification behavior.
• Structured reporting: Produce evaluation results in formats such as JSON and Markdown, supporting inspection and further analysis.
• Governance-oriented workflow: Organize policy evaluation around evidence, review, and deliberate decisions rather than treating every proposed change as automatically acceptable.

The policy conditions can cover relevant merge request metadata, including file patterns, line thresholds, labels, test status, pipeline status, scan status, and configured exceptions.

The current evaluation dataset uses synthetic merge request examples to provide a repeatable testing environment. These examples allow the engine to demonstrate policy behavior and comparison workflows without implying that synthetic results establish universal real-world accuracy.

The central idea is straightforward: a policy should not be judged only by how its rules look. It should also be evaluated by the decisions those rules produce.

How we built it

We designed PolicyProof around a deterministic evaluation engine and a structured policy definition model.

1. Express policies as data

Policies are represented in YAML rather than being buried entirely in application logic. This makes rule definitions easier to inspect, review, version, and change. Conditions describe which aspects of a merge request should influence an evaluation.

2. Evaluate against historical examples

The evaluation engine applies the configured policy to a dataset of merge request examples. With a fixed policy and fixed inputs, the deterministic approach supports repeatable results, making it easier to investigate unexpected decisions and compare alternative configurations.

3. Compare the baseline with the proposal

Backtesting becomes more useful when a proposed policy can be compared with an existing one. PolicyProof is designed to expose changes in decisions so that reviewers can investigate whether a change improves the intended behavior or introduces new problems.

4. Measure the outcomes

Policy evaluation should be more informative than a simple pass-or-fail result. Precision, recall, F1 score, and confusion-matrix information provide a more structured view of classification behavior. These metrics help reveal trade-offs that may otherwise remain hidden behind a single summary status.

5. Make the evidence inspectable

JSON and Markdown reports make evaluation results easier to inspect, preserve, and use in downstream workflows. Structured outputs also provide a foundation for future integration with automated review and governance systems.

6. Keep people accountable

PolicyProof follows the Assisted Autonomy approach. Automation performs repeatable evaluation and prepares evidence; a human remains responsible for interpreting the results and making consequential policy decisions. This distinction matters because a technically consistent evaluation does not automatically mean a policy is appropriate for every organization or context.

The project is developed in GitLab, with its source code, policy configuration, tests, documentation, and available CI/CD execution evidence providing a basis for technical review.

Challenges we ran into

One of the biggest challenges was turning policy evaluation into a meaningful engineering problem rather than simply checking whether a YAML file was syntactically valid.

A policy can be valid configuration and still produce undesirable outcomes. That meant the important question was not just whether a rule could be parsed, but how its behavior changed across representative examples.

The first challenge was reproducibility. Meaningful comparisons require consistent inputs and evaluation behavior. A deterministic engine and repeatable dataset help isolate the effects of a policy change instead of confusing those effects with unrelated variation.

The second challenge was evaluating trade-offs. A policy may catch more undesirable cases while also blocking more legitimate ones. Looking at precision, recall, F1 score, and confusion-matrix information provides a more useful basis for examining these trade-offs than relying on one aggregate number.

The third challenge was interpreting regressions. A changed decision is not automatically a defect. Some changes are intentional; others may indicate an overly broad condition, a missed exception, or an unexpected interaction between rules. The evaluation process therefore needs to surface changes for investigation rather than assume that every difference is inherently good or bad.

The fourth challenge was preserving the right balance between automation and control. Automated systems are useful for consistent evaluation, but governance decisions often depend on context that cannot be reduced to a metric alone. PolicyProof's Assisted Autonomy positioning keeps evidence generation separate from the responsibility to approve consequential changes.

Finally, integrating a local policy engine into a broader agent-driven development workflow introduces additional requirements around orchestration, permissions, integration contracts, and reliable execution. These are important engineering boundaries: the existence of a working evaluator does not, by itself, prove that an external agent integration is operational.

Accomplishments that we're proud of

We are proud of building a project around a concrete and often overlooked engineering principle: governance policies deserve regression testing too.

Our key accomplishments include:

• Building a deterministic policy evaluation approach using structured YAML definitions.
• Creating a reproducible evaluation environment using synthetic merge request examples.
• Supporting baseline and proposed policy comparisons to examine changed decisions.
• Incorporating established classification metrics into policy evaluation.
• Making evaluation results available through structured reports.
• Framing policy changes as testable behavioral changes instead of treating configuration edits as inherently safe.
• Designing the workflow around Assisted Autonomy, with human accountability retained for consequential decisions.
• Connecting the project's purpose to the broader post-code software delivery lifecycle.

These accomplishments establish a foundation for more rigorous policy testing. They also create a clear path for further development: richer evaluation datasets, improved explanation of individual decision changes, stronger regression controls, and verified integration with broader CI/CD and agent workflows.

We believe the most valuable outcome is not a claim that every governance problem has been solved. It is a concrete engineering foundation for investigating policy behavior before that behavior affects real delivery decisions.

What we learned

Building PolicyProof reinforced several important lessons.

First, configuration correctness and behavioral correctness are different. A rule can be valid, readable, and successfully executed while still producing an undesirable result. Testing the outcomes is essential.

Second, comparisons provide context that isolated scores cannot. A metric can summarize behavior, but a baseline-versus-proposed comparison helps explain what changed and which cases deserve attention.

Third, reproducibility is a governance advantage. When evaluations can be repeated against the same inputs, teams gain a stronger basis for debugging, review, and regression investigation.

Fourth, synthetic data is useful for controlled testing, but its limits must be acknowledged. Synthetic examples can validate logic and illustrate evaluation workflows; they cannot independently establish real-world effectiveness. Representative production data and carefully designed validation would be needed to make stronger claims about operational performance.

Finally, autonomy is not the same as removing human responsibility. In an Assisted Autonomy workflow, automation can perform repetitive checks and prepare evidence while people retain authority over decisions that require context, risk assessment, or organizational judgment.

These lessons shaped both the implementation and the way we think about deploying policy automation responsibly.

What's next for PolicyProof: Evidence Before Enforcement

Our next steps focus on making policy evaluation more explainable, representative, and useful in real development workflows.

More representative evaluation

We want to expand beyond synthetic examples with carefully curated, representative datasets and clearer validation procedures. This would help teams understand how evaluation results transfer to their own repositories and governance requirements.

Better explanations for changed decisions

A reviewer should be able to understand not only that a decision changed, but which conditions contributed to that change and why it matters. More detailed, case-level explanations would make regression investigation faster and more actionable.

Stronger governance controls

Further development could introduce configurable promotion thresholds, explicit review requirements, policy version history, and stronger safeguards around changes that exceed an organization's accepted risk limits.

Verified CI/CD integration

We want to connect policy backtesting more directly to development workflows so that teams can run evaluations when policy definitions change, inspect the results, and use verified checks to inform promotion decisions.

Agent-assisted workflows

A future integration with GitLab Duo Agent Platform could help users draft policy proposals, explain evaluation findings, and prepare review summaries. Such capabilities would need to be implemented and verified independently; the deterministic policy engine alone should not be mistaken for a completed agent integration.

Broader lifecycle support

We aim to strengthen PolicyProof's role across Verify, Secure, Release, Configure, Monitor, and Govern, with each lifecycle connection backed by a clear implementation, meaningful evidence, and appropriate safeguards.

Our long-term vision is a governance workflow in which policy changes are proposed transparently, evaluated reproducibly, compared against a baseline, reviewed by the right people, and promoted only through an accountable process.

Closing principle

Software delivery has become increasingly automated, but the rules governing delivery also change—and those changes can introduce risk.

PolicyProof brings a testing mindset to that overlooked layer. By evaluating policies against historical examples, comparing decisions, measuring outcomes, and supporting human review, it aims to make governance decisions more deliberate and evidence-driven.

PolicyProof — test the policy, understand the consequences, and put evidence before enforcement.

Autonomy Track: Assisted Autonomy.

Built With

• assisted-autonomy
• automation
• backtesting
• ci-cd
• confusion-matrix
• counterfactual-analysis
• deterministic
• devsecops
• duo-agent-platform
• evaluation
• f1-score
• gitlab
• gitlab-ci-cd
• gitlab-duo
• governance
• json
• machine-learning
• markdown
• metrics
• policy-as-code
• precision-recall
• python
• regression-testing
• testing
• yaml