Inspiration
Business policies are written in natural language. However, when multiple rules come into play the outcome can be unpredictable. For example, one rule may define a refund policy enabling a customer to ask for a refund if they cancel a replacement. However, another rule may define a 30 day rule that states no refunds can be given.
We wanted to create a tool that would identify rule conflicts within business policies.
This inspired us to create PolicyForge; an AI-assisted CI/CD testing environment for business policies.
Overview
PolicyForge converts business policy text to structured rules that enable teams to test how business policies work.
Key features include:
AI assisted policy compiler: Using NVIDIA Nemotron via Nebius Token Factory, natural language text can be converted into structured rules.
Rule validation: Policy rules can be validated using Pydantic schemas and source tracking.
Customer Journey Simulation: Policy rules can be simulated using events such as delivery, requesting a replacement, cancelling and requesting a refund to see which rules will apply.
Conflict Detection: Identify conflicting rules.
Policy Gap Detection: Find scenarios that have been tested but do not match any applicable rule.
Counter-example minimisation: Minimise the customer journey to find a simpler customer journey that still finds the issue.
Stress Testing: Create deterministic edge cases and test policy rules.
What if: Run the same scenarios on two different policy versions to see how the outcome changes.
Evidence Reporting: Report analysis of rules using sources in markdown and JSON format.
For this demonstration we created a fictional e-commerce business called NovaCart with refund, replacement and cancellation policies.
Within one of the scenarios there is a conflict. The customer cancels their replacement and requests a refund on day 31. One rule enables the refund, however another rule disables refunds after 30 days. PolicyForge can identify the conflicting rules and give a customer journey that can be reproduced.
Implementation
PolicyForge works on the principle that the AI will interpret the policy and then deterministic code will validate it.
NVIDIA AI: We have integrated NVIDIA Nemotron via Nebius Token Factory. Policy can be compiled using this component, however it is optional. The human readable policy rules can be converted into structured rules with decisions, triggers, conditions, impacts and source evidence.
Backend: We created a python FastAPI backend. We used Pydantic v2 to validate requests and structure the rules. Deterministic simulation is used to process events in chronological order, update the order status and identify matching rules and decisions.
Validation Engine: Conflicts can be identified. Policy gaps can be found. Scenarios can be run, edge cases identified, counter-examples minimised and decisions compared.
Frontend: We created an interactive interface using Next.js, React, TypeScript and TailwindCSS. Rules can be viewed, customer journeys can be simulated, findings explored, stress tests run, policy versions can be compared and reports exported.
GitHub was used to manage version and CI. Backend and Frontend were automatically tested. The application was hosted using Render.
The public demo has the AI compiler turned off so judges can explore the deterministic validation features. However, the NVIDIA compiler has been turned on but requires authorisation.
Challenges
Ambiguity in rules
Policies written in natural language can be ambiguous with missing conditions or unclear time frames. To reduce this ambiguity rules need to be structured, source tracked and manually reviewed.
Conflicting rules
Multiple rules can be applicable to a decision but give different outcomes. We explicitly decided on what the decision should be and reported any conflicts.
Understanding complex conflicts
Long sequences of events can make it hard to understand why a conflict has occurred. We implemented counter-example minimisation to simplify the customer journey.
Meaningful evidence
A test may pass but not cover all conditions. We have reported what was covered during testing.
Secure AI inference
Explicit consent was added, access was limited and budgets were set up. This allowed us to separate the AI powered policy compiler from the deterministic public demo.
Achievements
Created a working end to end application to validate policies.
Integrated an AI policy compiler using NVIDIA Nemotron. We had previously validated a real provider extraction for a simple refund policy.
Found and reproduced a real conflict in our NovaCart demo.
Identified a policy gap and minimised the reproduction of the issue to one event.
Successfully ran a stress test with 30 deterministic edge cases.
Compared NovaCart v1 and v2 using 5 common journeys. We identified 2 customer decisions that changed from ALLOW to DENY.
Added source tracking explanations, traceability and downloadable reports.
Validated the main workflows in the public demo using browser testing.
Learning
Not all AI applications require AI to make the final decision.
Language models are great at interpreting human written requirements however deterministic systems provide consistency, repeatability and explainability.
Typed interfaces, validation, repeatable testing, careful API integration, secure credentials and giving control to the user to activate the policy were identified as important.
Future Improvements
Adversarial scenarios can be generated using AI.
More policy types can be supported such as subscriptions, insurance, finance and service level agreements.
Policy versions can be stored and reviewed.
Policy regression can be detected using GitHub pull requests.
More semantic validation can be performed and policy coverage identified.
Stronger authentication can be used, quotas can be made more durable and audit history can be added.
Ultimately we want policy validation to become as standard as testing software.
PolicyForge - Test your business policies before they fail.
Built With
- fastapi
- github-actions
- javascript
- nebius-token-factory
- next.js
- nvidia-nemotron
- pydantic
- python
- react
- render
- tailwind-css
- typescript
