CaseGraph — PayPal Exception CaseGraph

Turn messy payment disputes into auditable, provenance-gated evidence graphs before money moves

Comment

Inspiration

When a payment dispute lands, a merchant faces three pressures at once: a strict response deadline—roughly 10 days for an internal inquiry and 20 to 45 days for an external chargeback (Solidgate)—a pile of fragmented customer chats, carrier scans, and refund logs (Merchant Risk Council), and a financial API where a wrong call moves real money.

While researching the dispute-automation market, we noticed two critical gaps:

  • Black-box representment vs. auditability: Incumbent chargeback platforms (Chargeflow, Justt, Disputifier, and Stripe Smart Disputes) optimize for automated representment volume on card networks, leaving merchants unable to audit why a specific claim was asserted. Meanwhile, PayPal's Resolution Center and Disputes v1 API provide rich dispute primitives but no native AI evidence builder.
  • Unverified LLM prose is unsafe for financial writes: Recent financial root-cause analysis research (FinRCA-Bench, arXiv:2608.18534) shows that holding the LLM fixed while enforcing typed provenance graph retrieval moves required-record recall from 0.83% to 77.70% and exact accuracy from 2.05% to 72.44%, while strict returned-evidence contract accuracy on raw models is only 5.72%. Similarly, empirical studies on agentic fraud investigation (Toward Auditable Fraud Detection, arXiv:2607.19266) demonstrate that a free-form LLM agent can underperform a direct classifier (65.0% vs. 71.7%) while writing a convincing rationale every time.

We built CaseGraph to bridge that gap around one non-negotiable invariant: AI interprets evidence. PayPal remains the financial source of truth. Humans authorize consequential actions.

What it does

CaseGraph transforms a messy PayPal dispute into a forensic investigation dossier and provenance-gated execution workflow:

  • Syncs authoritative PayPal dispute state: Connects to the PayPal Disputes v1 API and verified webhook stream (/v1/notifications/verify-webhook-signature) to ingest dispute amounts, reason codes, lifecycle stages, and allowed actions derived strictly from PayPal's HATEOAS links (send-message, provide-evidence, appeal, make-offer, accept-claim, escalate, provide-supporting-info, and acknowledge-return-item).
  • Versions immutable evidence and hash-bound files: Ingests customer/merchant transcripts, support notes, and malware-scanned multipart evidence attachments (PDF, JPEG, PNG, GIF) as append-only, SHA-256-hashed evidence set revisions.
  • Compiles structured AI output into a byte-checked epistemic graph: Freezes the case and evidence revisions into a canonical JSON bundle, queries a structured LLM provider (OpenAI /v1/responses or DeepSeek /v1/chat/completions, with a deterministic reference reasoner for zero-credential demos), and compiles the output (validate_ai_output) into seven typed node classes (VERIFIED_FACT, CUSTOMER_CLAIM, MERCHANT_CLAIM, AI_INFERENCE, MISSING_EVIDENCE, ACTION, EVENT) and eight provenance relations (SUPPORTS, CONTRADICTS, DERIVED_FROM, MISSING_FOR, PRECEDES, REFERENCES, RESOLVES, CLAIMS).
  • Enforces a deterministic safety policy over provenance: Walks grounding chains from recommended actions back to PayPal facts and evidence revisions, blocking ungrounded inferences, blocking high-risk financial actions supported by missing evidence, and downgrading recommendations when supporting nodes are contradicted.
  • Evaluates negation-aware dispute readiness: Maps all eight PayPal dispute reasons to required evidence items (present, missing, contradicted, unknown) with clause-level negation detection—so a note stating "no delivery proof was provided" counts against fulfillment rather than satisfying it—without ever presenting coverage as a fabricated win-rate probability.
  • Binds human approval to single-write execution and fail-closed reconciliation: Requires an authorized human to approve the exact typed action payload and file SHA-256 hashes (approval_hash), re-validates live PayPal state immediately before sending, executes at most once, and locks any network timeout or 5xx response into UNKNOWN_OUTCOME until reconciliation confirms what happened on PayPal's servers.

How we built it

  • PayPal Integration & Zero-Credential Simulator (backend/paypal/): Implemented full OAuth2 client-credentials authentication (auth.py), all 11 read/write/verification operations in the PayPal Disputes v1 API (disputes.py, http.py, webhooks.py), and a stateful simulated PayPal sandbox (simulator.py, simulation_data.py) inspired by PayPal Dispute API test values. Every simulated payload is stamped "simulated": true, supports fault injection (CASEGRAPH_SIMULATED_FAULT=timeout|server_error), and is hard-blocked from starting when CASEGRAPH_ENV=production.
  • Epistemic Graph Compiler & Safety Engine (backend/casegraph/, backend/safety/): Informed by provenance-gated agent execution research (Agent-Sentry, Janus, PROVISO, and HOPM), validate_ai_output enforces that every VERIFIED_FACT statement is byte-identical to the persisted PayPal fact value, every claim excerpt is a verbatim substring of the cited evidence revision, speakers match the transcript turn, and provenance edges form a valid DAG.
  • Multi-Provider AI Reasoning Pipeline (backend/ai/): Built a frozen-bundle reasoning service (AIReasoningService) with idempotent request claiming, stale-input detection if evidence changes mid-inference, and dual API adapters supporting OpenAI Structured Outputs (/v1/responses), OpenAI-compatible Chat Completions (/v1/chat/completions for DeepSeek and JSON-mode gateways), and a deterministic reference reasoner (rules.py).
  • Durable Queue, Locking & Reconciliation (backend/work/, backend/db/): Built on FastAPI, SQLAlchemy 2.0, and Alembic (20261108_0010), featuring a lease-based background worker (scripts/worker.py) with exponential backoff, dead-lettering, PostgreSQL per-dispute advisory locks, and automatic/manual reconciliation (reconciliation.py).
  • Three-Pane Forensic Dossier UI (frontend/): Designed a React + TypeScript investigation workspace inspired by Palantir Gotham, Sayari Graph, Bloomberg, and Hoefler&Co tabular typography—combining a deadline-sorted CaseDocket, an interactive SVG GraphCanvas, a provenance/readiness Inspector, an exportable audit ExplanationPanel, and a typed ActionComposer.

Challenges we ran into

  • Preventing LLM fact laundering and prompt injection: Early tests showed that untrusted customer transcripts containing prompt-injection instructions (or subtle paraphrasing) could coax a raw model into citing a customer statement as a verified fact or recommending an unauthorized refund. We solved this by separating PAYPAL_FACT and EXTERNAL_EVIDENCE at the schema level and enforcing byte-exact fact matching and verbatim substring verification in backend/casegraph/validate.py.
  • Eliminating false positives in evidence readiness checklists: Naive keyword matching initially treated "IP address" as proof of delivery address and counted "no delivery proof was provided" as present fulfillment evidence. We tightened signal patterns and built a clause-window negation detector (_NEGATION) plus graph-contradiction downgrades (status: "contradicted") in backend/services/readiness.py.
  • Handling ambiguous outcomes on non-idempotent financial writes: Because PayPal dispute actions move money or alter legal dispute state, retrying a timed-out POST /make-offer or POST /provide-evidence risks double-submission. We engineered ActionEngine to classify explicit 4xx errors as FAILED while isolating timeouts, transport drops, and 5xx responses into UNKNOWN_OUTCOME, blocking any new approval or write on that dispute until reconciliation proves the remote outcome.
  • Bridging OpenAI Responses API and Chat-Completions providers: While OpenAI's /v1/responses endpoint enforces strict JSON schemas natively, many OpenAI-compatible providers (such as DeepSeek) expose /v1/chat/completions. We built an automatic mode router in backend/ai/providers/openai.py that supports both APIs while running the exact same local AIOutput and validate_ai_output compiler gates.

Accomplishments that we're proud of

  • End-to-end provenance from PayPal bytes to human-approved action: Every recommendation in CaseGraph can be traced step-by-step—via the visual GraphCanvas or the deterministic GET /api/cases/{id}/ai/analyses/{uid}/explain audit artifact—back to the exact PayPal fact revision or verbatim transcript quote.
  • 212 passing automated tests and a 60-case adversarial benchmark: Our verification suite (212 passed, 0 failed, 0 skipped) covers ASGI HTTP journeys, hash-bound multipart file uploads, OIDC JWKS rotation, and reconciliation locks. In addition, our 60-scenario benchmark (scripts/benchmark.py across 24 dev, 16 holdout, and 20 redteam cases including customer/merchant prompt injections, fabricated evidence, and HATEOAS mismatches) verifies 0.0% safety violations.
  • Two-minute, zero-credential judge experience: By shipping a production-forbidden simulated PayPal sandbox (CASEGRAPH_PAYPAL_TRANSPORT=simulated) and scripts/demo_seed.py, any judge can clone the repository and experience the entire dispute investigation, contradiction downgrade, human approval, and fault-injected UNKNOWN_OUTCOME reconciliation flow in under two minutes without needing API keys.
  • Editorial dossier aesthetic verified across 6 viewports: Replaced generic SaaS dashboard cards with a high-contrast warm-paper forensic dossier that passed headless Chrome DevTools audits across 6 viewports (1440x900 down to 390x844) with zero horizontal overflow and WCAG AA contrast ratios.

What we learned

  • Compiler invariants beat prompt engineering for financial AI: Telling an LLM in a system prompt "only use authoritative PayPal state and allowed actions" fails under adversarial input. Enforcing HATEOAS link intersection, refund-ceiling bounds (refund_details.allowed_refund_amount), and byte-exact quote validation in deterministic Python code makes unsafe model outputs unrepresentable.
  • Post-approval substitution is a real threat surface: Studying Janus (arXiv:2609.38266) reinforced why "human-in-the-loop" is meaningless if the underlying state shifts between approval and execution. Binding case_revision, evidence_set_revision, analysis_uid, safety_uid, policy_hash, and payload_hash into a single approval_hash—and re-fetching live PayPal state immediately before the write—closes that window.
  • Evidence readiness must be honest about its epistemic limits: Merchants need to know which document is missing for a given PayPal reason code, not an uncalibrated "win probability" percentage. Explicitly labeling our readiness panel's basis and testing that no probability claims leak into the payload kept the product honest.

What's next for CaseGraph — PayPal Exception CaseGraph

  • Typed provenance graph retrieval for high-volume enterprise cases: As merchant dispute histories grow to hundreds of evidence revisions and prior orders, we plan to implement default-deny typed graph traversal over historical transaction relationships, following the architecture validated by FinRCA-Bench.
  • Controlled live-product operator study: Our repository already ships a counterbalanced 20-scenario within-participant study framework (validation/, scripts/serve_validation_study.py, and scripts/analyze_validation_study.py); next, we will run confirmatory operator trials to measure time-to-decision and contradiction-detection accuracy under our hash-locked protocol (docs/VALIDATION_PROTOCOL.md).
  • Automated carrier & ERP evidence connectors: Expanding EvidenceService with read-only connectors for carrier tracking webhooks and order-management invoices so missing fulfillment or refund proof can be staged automatically as candidate evidence revisions for human review.

Built With

Share this project:

Updates

Submission history