PayPal PolicyTwin

Inspiration

A financial policy can look safe in plain English and still cause dangerous behavior when it reaches real payment systems.

For example, a policy such as “automatically refund transactions of $50 or less when tracking is unavailable” leaves several questions unanswered:

Should digital products be refunded even though they never have tracking?
What happens when a dispute is already open?
Can a retried request create a second refund?
What happens when the same webhook is delivered twice?
Does the $50 threshold apply only to USD?

Traditional tests verify whether software behaves as written. They do not always reveal whether the written policy itself is incomplete or unsafe.

That inspired PayPal PolicyTwin: a counterfactual financial-policy testing system that changes exactly one transaction fact at a time and observes whether the financial outcome changes correctly.

The core principle is:

AI proposes → Code validates → Simulator or Sandbox executes → Code verifies → Human decides

PayPal PolicyTwin is an independent hackathon project designed around PayPal payment workflows. It is not an official PayPal internal system.

What it does

PolicyTwin compares a current policy with a proposed policy and generates matched transaction twins—two synthetic transactions that are identical except for one meaningful field.

Examples include:

Physical product versus digital product
No dispute versus an open dispute
First request versus a retried request
Original webhook versus duplicate webhook
USD versus EUR
Amount below, at, or above the refund boundary

The system then:

Converts both policies into validated, structured rules.
Maps the payment products, states, and operations affected by the change.
Generates core and adversarial transaction twins.
Rejects any pair that changes more than one causal field.
Evaluates the current and proposed policies against every scenario.
Replays payment operations using a deterministic PayPal simulator.
Checks eight financial safety invariants.
Produces a PASS, REVIEW, or BLOCK recommendation.
Explains every failure using evidence from the tested scenarios.
Proposes a corrected policy while preserving the original business objective.
Requires a human to approve the repair, rerun every scenario, and make the final release decision.

In the guided demonstration, Version A sends all missing-delivery claims for manual review. Version B automatically refunds transactions of $50 or less when tracking is unavailable.

PolicyTwin generates 10 transaction-twin pairs, representing 20 scenarios. The proposed policy is BLOCKED after detecting:

Automatic refunds for digital products
Automatic refunds while a dispute is open
Multiple refunds caused by request retries
Duplicate operations and state regression from repeated webhooks
Ambiguous behavior across currencies

The Policy Repair agent proposes Version C with explicit rules for product type, currency, disputes, retries, and duplicate events. After a human approves the repair for testing, PolicyTwin reruns every scenario. Version C passes all 10 pairs with zero critical findings and zero invariant failures.

A PASS is still only a recommendation. PolicyTwin never releases or approves a policy automatically.

How we built it

We built PolicyTwin as a full-stack application using Next.js, React, TypeScript, Zod, PostgreSQL, AG Grid, and Render.

The system contains six specialized agent roles:

Policy Reader converts policy text into structured rules.
Policy Impact Mapper identifies affected PayPal products, states, and required tests.
Twin Generator creates matched scenarios that differ in exactly one field.
Adversarial Scenario Agent searches for boundaries and failure conditions.
Impact Analyst explains the detected financial consequences.
Policy Repair Agent proposes a safer policy without changing its intended business objective.

When configured, the agents can use OpenAI Structured Outputs. Without an API key, deterministic agents implement the same contracts, allowing the demo to remain reproducible and functional.

AI output is never trusted directly. Every output passes through schemas, deterministic guards, causal-field validation, and financial invariants. The AI can propose scenarios and explanations, but it cannot change a verdict or approve a release.

The full test suite runs through a deterministic in-process PayPal simulator so retries, duplicate webhooks, out-of-order events, and failures can be reproduced safely.

We also implemented a separate PayPal Sandbox adapter using PayPal REST APIs for:

OAuth authentication
Creating and retrieving orders
Confirming and capturing payments
Authorizing and voiding payments
Issuing refunds
Reusing PayPal-Request-Id values for idempotent retries
Verifying webhook signatures

The two execution lanes are deliberately separated:

Lane 1 runs every scenario using the deterministic simulator and determines the release recommendation.
Lane 2 can verify one user-selected pair using the configured PayPal Sandbox adapter. It attaches external evidence but never silently changes the Lane 1 verdict.

The interface clearly labels whether evidence is SIMULATED or produced by PAYPAL SANDBOX. The public guided demo currently shows simulated execution and never presents synthetic results as real PayPal data.

AG Grid powers the Transaction Twin Matrix, where users can search, filter, compare, inspect, and export scenario evidence. The Scenario Inspector displays the changed field, policy decision path, payment operations, invariant results, and evidence provenance.

Challenges we ran into

The largest challenge was allowing AI to contribute without allowing it to become the financial decision-maker. We solved this by treating AI output as an untrusted proposal and placing deterministic validation around every agent.

Another challenge was guaranteeing that each transaction twin changed exactly one meaningful fact. A multi-field comparison might reveal a difference, but it would not prove which field caused it. We created a causal-field catalog and validator that rejects invalid pairs.

Payment retries and webhooks introduced additional complexity. A system may make the correct policy decision but still move money twice. We therefore tested idempotency, duplicate-event handling, event ordering, and local-state regression as financial invariants—not merely implementation details.

We also needed to communicate provenance honestly. The hosted guided run uses fictional policies, synthetic data, and simulated payments. The interface displays those limitations throughout the experience instead of hiding them.

Finally, we had to make complex evidence understandable within minutes. We designed the experience around a guided story: unsafe proposal, BLOCK recommendation, explainable evidence, human-approved repair, complete rerun, and PASS recommendation.

Accomplishments that we're proud of

We are proud that PolicyTwin is more than a policy chatbot or static dashboard. It runs an end-to-end, evidence-producing release workflow.

Major accomplishments include:

Generating matched transaction twins with single-field causality
Detecting policy ambiguity and unintended expansion
Testing 20 scenarios and eight financial invariants in the guided run
Reproducing retries, duplicate webhooks, and state-regression failures
Producing a complete BLOCK → repair → rerun → PASS journey
Preserving a strict human decision boundary
Separating simulated evidence from PayPal Sandbox evidence
Providing searchable scenario evidence through AG Grid
Building resumable run stages, access controls, rate limits, sanitization, and audit events
Creating unit, integration, evaluation, accessibility, and end-to-end test coverage

Most importantly, PolicyTwin does not simply say that a policy is dangerous. It shows which transaction fact caused the unexpected result, which invariant failed, and how the proposed repair changes the outcome.

What we learned

We learned that AI is most useful in financial systems when it generates possibilities and explanations while deterministic software enforces safety.

We also learned that policy correctness and payment correctness cannot be separated. A policy can produce the right decision while retries or duplicate events still move money incorrectly.

Counterfactual testing made failures easier to explain. Instead of presenting an abstract risk score, PolicyTwin can say: “These transactions were identical except for the dispute state, but both were automatically refunded.”

Finally, we learned that trustworthy financial AI requires visible provenance. Users must always know what was generated by AI, what was verified by code, what was simulated, and what still requires human judgment.

What's next for PayPal PolicyTwin

Our next steps are to:

Complete repeatable live validation against PayPal Sandbox and attach sanitized evidence
Validate the OpenAI Structured Outputs path using a larger policy evaluation dataset
Add PayPal JavaScript SDK or Hosted Fields for a production-appropriate payment-source flow
Expand beyond missing-delivery refunds to disputes, subscriptions, payouts, authorization policies, seller protection, and cross-border payments
Integrate PolicyTwin into policy-authoring and CI/CD workflows as a pre-release gate
Add organization-level authentication, roles, approvals, and shared rate limiting
Support larger policy datasets and historical anonymized test cases
Add downloadable PDF evidence reports and policy-version comparison history

Our long-term vision is for every high-impact financial policy change to receive a counterfactual safety test before it reaches real customers or real money.

Live demo: https://policytwin.onrender.com

Demo video: https://youtu.be/H9wZcyeDpGk