Inspiration

What if the most obvious explanation for a financial incident is the wrong one?

Imagine you're managing payment operations for an e-commerce business. Refunds suddenly surge. Your dashboards point toward mobile transactions as the biggest contributor. The numbers look convincing, and the pressure to act is mounting.

But mobile isn't necessarily the problem.

Behind that apparent pattern could be a failing delivery partner, a defective product batch, or several overlapping issues that conventional dashboards struggle to untangle.

Acting on the wrong explanation can cost a business twice: once through an ineffective response, and again because the actual problem continues unchecked.

With AI increasingly entering financial operations, we wanted to tackle a fundamental question:

How can we make AI powerful enough to investigate financial incidents, but trustworthy enough to influence what happens next?

That's why we built PayOps Sentinel — an AI financial incident commander that doesn't just generate explanations. It actively challenges them before anyone touches money.

What it does

PayOps Sentinel transforms fragmented payment investigations into a unified, evidence-driven workflow:

Detect → Investigate → Challenge → Quantify → Recommend → Authorize → Verify.

Detect and investigate

When refund activity deviates significantly from its historical baseline, PayOps Sentinel automatically identifies the anomaly.

Claude investigates the incident through read-only, argument-validated tools, examining fulfillment partners, product SKUs, categories, regions, and payment channels. Rather than accepting the first plausible explanation, it explores competing hypotheses.

Challenge the AI before trusting it

This is where PayOps Sentinel is different.

Every proposed driver must survive an independent, deterministic hypothesis-verification engine.

The engine evaluates sample sizes, exact statistical tests with multiple-testing correction, stratified Mantel-Haenszel comparisons, aliasing, and conditional re-testing for overlapping contributors.

The AI cannot verify its own conclusions.

The server independently evaluates every proposed driver, distinguishing supported associations from contradicted hypotheses, insufficient evidence, and unresolved explanations.

When the evidence cannot distinguish two possibilities, PayOps Sentinel abstains instead of guessing.

Turn evidence into business decisions

Identifying a suspicious pattern is only the beginning.

PayOps Sentinel calculates affected transaction cohorts, financial exposure, refunded amounts, remaining refundable value, and outstanding customer cases.

Every financial figure is explicitly labeled as actual PayPal data, synthetic history, or a scenario estimate.

The system then recommends targeted operational responses, such as investigating a fulfillment partner, reviewing a defective product batch, or prioritizing customer outreach. These are clearly identified as manual follow-up actions.

Take action without giving AI control of money

For legitimate customer refund cases, Claude can draft a structured proposal — but it cannot execute it.

Every refund must pass:

  • 16 deterministic financial policy checks.
  • A second operator's signed approval bound to the exact action.
  • A single-use execution claim.
  • Fresh PayPal validation.
  • Idempotent sandbox execution.
  • Independent verification through PayPal.

The investigation, operational response, and financial execution remain separate. Successfully refunding one customer does not mean the broader incident has been resolved.

See it in action

In our live demonstration, the merchant's refund rate rises from 3.6% to 26.8%.

A conventional share-based approach identifies mobile transactions as the leading suspect.

PayOps Sentinel challenges that conclusion. It discovers that 70% of mobile orders use NorthPost, compared with 46% of orders overall. Mobile fails the corrected statistical test, while NorthPost and the HDPH-200 headphone batch emerge as supported contributing factors.

The platform identifies the affected cohorts, quantifies their financial exposure, and recommends targeted interventions.

It then processes a legitimate $12 refund through the real PayPal Sandbox, with separate authenticated operators and verified execution.

Finally, when a malicious customer note attempts to manipulate the AI into issuing a $500 refund, Claude refuses the instruction. A manually submitted version of the malicious proposal is independently blocked by the financial policy engine.

The result: AI-powered investigation, evidence-backed decisions, and controlled financial execution — all in one workspace.

How we built it

We designed PayOps Sentinel around one architectural principle:

Let AI investigate. Let evidence challenge it. Let deterministic controls govern money.

  • Backend: FastAPI, SQLAlchemy, and PostgreSQL, with an idempotent financial ledger using integer minor units and an append-only audit log.
  • AI reasoning: Claude through a manually orchestrated tool-calling loop, with validated arguments, structured outputs, recorded evidence, and independently checked hypotheses. The video demonstrates live model execution; an explicitly labeled replay mode is available for judges without an API key.
  • Statistical verification: An independent hypothesis-verification engine using exact tests, multiple-testing correction, stratified comparisons, and conditional hypothesis testing.
  • PayPal integration: Real sandbox captures, capture-refund APIs, idempotency keys, provider read-back, verified webhooks, and the PayPal Agent Toolkit in strictly read-only mode.
  • Frontend: Next.js, AG Grid Enterprise 36, and AG Studio 3, creating an interactive financial investigation environment.

AG Studio: an intelligent workspace for financial operations

AG Studio is not simply a visualization layer. It is where operators interact with the investigation.

We built two integrated canvases: an Operations Overview and a dedicated Incident Investigation workspace.

Three custom widgets — Evidence Inspector, Investigation Timeline, and Policy Decision — integrate alongside Studio's built-in charts and grids. Operators can compose their own layouts, inspect supporting evidence, and explore the ledger without leaving the workspace.

Studio's coordinated filters dynamically re-slice the displayed data. In our demonstration, filtering to a single carrier changes the displayed refund rate from 10% to 19%.

We also integrated AG Studio's AI assistant with Claude through a custom LLM adapter and read-only tools, allowing operators to ask questions about an incident and receive answers grounded in stored statistical results and ledger records.

Challenges we ran into

Our biggest engineering breakthrough came from discovering that our own approach was wrong.

Initially, we believed that verifying the numerical evidence behind an AI-generated explanation was enough.

An adversarial audit proved otherwise.

Because shares of excess refunds are normalized, an unrelated but overlapping segment could satisfy our threshold and be incorrectly labeled as a cause.

We realized we weren't solving the hardest part of the problem.

An AI can cite a correct number and still reach the wrong conclusion.

So we redesigned the investigation workflow around competing hypotheses, independent statistical verification, and explicit uncertainty.

A subsequent independent review identified another weakness: our statistical tests were overconfident at small sample sizes.

We responded by introducing exact tests, correcting for multiple comparisons, and evaluating the resulting engine on held-out synthetic scenarios.

Instead of hiding our failures, we used them to build a more trustworthy system.

Accomplishments that we're proud of

Building an investigator that knows when not to answer

Across 360 held-out synthetic evaluation runs per system, our deterministic hypothesis-verification engine achieved:

Evaluation metric Share-based baseline PayOps Sentinel engine
Incorrect driver attribution 26.9% 2.8%
Correct abstention when evidence was inconclusive 0% 98%
Primary-driver identification 87.7% 88.3%

There is an important trade-off: the engine abstained in 9.3% of runs where a driver existed.

In financial operations, we believe that uncertainty should be visible rather than disguised as confidence.

Building a financial workflow where AI cannot bypass authorization

Our safety evaluation passed all 16 attack and failure scenarios, with zero unauthorized refunds.

We successfully demonstrated live Claude investigation, authenticated two-operator approval, real PayPal Sandbox refund execution, and independent provider verification.

Building beyond a proof of concept

PayOps Sentinel has a public, reproducible codebase, 158 passing backend tests, verified GitHub Actions workflows, and an end-to-end interactive incident workspace.

Most importantly, we connected AI reasoning to a meaningful business workflow: understanding financial incidents, assessing their scope, prioritizing responses, and safely managing individual payment actions.

What we learned

Building PayOps Sentinel changed how we think about AI in financial systems.

We learned that the goal should not be to make AI sound more confident.

It should be to make its conclusions more accountable.

On straightforward incidents, a well-designed SQL baseline can perform just as well as an AI investigator. The meaningful opportunity lies in investigating competing explanations, adapting the investigative approach, and communicating evidence that humans can inspect.

We also learned that trust is not a feature added at the end. It must be built into the architecture.

Our evaluation uses synthetic merchant histories, and financial execution runs in the PayPal Sandbox. These are important limitations, but they provide a reproducible foundation for future validation with real merchant data.

What's next for PayOps Sentinel

Our vision is to evolve PayOps Sentinel into a comprehensive financial incident-response platform for merchants and payment-operations teams.

Next, we want to introduce real carrier and inventory integrations, incorporate signals from PayPal's Disputes API, support multi-merchant environments, and validate our hypothesis engine against real merchant histories.

Ultimately, we want to help businesses move from reactive payment monitoring to proactive, evidence-driven financial operations.

Because when money is on the line, an intelligent answer isn't enough. It needs to be an answer you can trust.

Built With

Share this project:

Updates

Submission history