Inspiration
When a shopper pays via PayPal, money moves synchronously via API calls, while the merchant's order state depends on asynchronous webhooks[cite: 4, 5]. In real-world integrations, webhooks drop, arrive late, duplicate, or arrive out of order[cite: 4]. When a PAYMENT.CAPTURE.COMPLETED webhook is lost, a customer is charged while the merchant's store stays stuck on PENDING_PAYMENT—creating an "orphaned checkout" that merchants often only discover days later from angry customer complaints[cite: 4, 6, 31].
Existing tools leave an unaddressed seam between systems[cite: 5, 7]:
- PayPal's native dashboard tools inspect PayPal-side events without visibility into the merchant's order database[cite: 7].
- PayPal's Transaction Search API lags real time by up to ~3 hours, making it too slow for immediate checkout recovery[cite: 4, 6, 7].
- Generic webhook gateways retry delivery, but they are payment-agnostic and cannot reconcile PayPal's ledger against merchant fulfilment states[cite: 5, 7].
We built LedgerLoop (PAYPAL_AI) to close this gap: a payment-reliability layer that continuously diffs PayPal's record against the merchant's order state, heals mismatches safely, and proves its own resilience live through a built-in Chaos Lab[cite: 1, 6, 10, 40].
What it does
LedgerLoop closes the gap between what PayPal has recorded and what a merchant system believes happened[cite: 4]:
- Verified Webhook Ingest & Event Store: Captures exact raw webhook bytes and headers, verifies signatures via PayPal's
verify-webhook-signatureAPI, and deduplicates events using database-level unique constraints (event_id) before acknowledging with a200 OK[cite: 7, 11, 21, 22]. - Deterministic State Machine & Reconciler: Models every order through a strict state machine (
CREATED->APPROVED->CAPTURED->COMPLETED->FULFILLED, plusDENIED,REFUNDED, andDISPUTED)[cite: 9, 11]. An APScheduler worker polls PayPal's Orders v2 API (GET /v2/checkout/orders/{id}) for overdue orders and runs sliding 31-day sweeps via the Transaction Search API to catch untracked payments[cite: 7, 11, 13, 17]. - 8-Class Anomaly Detection & Idempotent Healing: Deterministically detects eight anomaly classes (
A1 MISSING_WEBHOOKthroughA8 UNKNOWN_PAYMENT)[cite: 9, 24] and executes idempotent merchant actions (such as fulfilling or revoking an order) using unique idempotency keys and optional human approval mode[cite: 9, 11, 13, 25]. Amount mismatches (A6) are escalated to humans and never auto-fixed[cite: 9, 24]. - Two-Lane Timeline & Grounded AI Explainer: Displays a side-by-side "PayPal says vs. Merchant says" timeline backed by raw event evidence, paired with an AI incident explainer that translates structured anomaly JSON into a plain-language 3-sentence summary[cite: 6, 11, 16, 24].
- Built-in Chaos Lab & Mock Mode: Injects live
DROP,DUPLICATE,DELAY, andREORDERfaults on incoming webhook streams and runs an automated 12-scenario resilience matrix, alongside an offline Mock Mode powered by recorded PayPal fixtures[cite: 4, 11, 16, 28].
How we built it
We architected LedgerLoop as a modular monolith with clear internal boundaries and swappable merchant adapters[cite: 17, 19]:
- Backend & Database: Built with Python FastAPI, Pydantic, SQLAlchemy, Uvicorn, and APScheduler (DB-backed in-process background worker), backed by PostgreSQL (Neon/Supabase) with SQLite support for local runs[cite: 17, 18].
- PayPal API Integrations: Integrated the PayPal JavaScript SDK (demo storefront checkout), Orders v2 API, Webhooks Management API (
verify-webhook-signature), and Transaction Search API viahttpx[cite: 7, 17, 18]. Adaptive SLA Engine: Computed dynamic webhook deadlines using rolling delivery latencies bounded between a 30-second floor and a 600-second cap[cite: 24]:
Frontend & Realtime Updates: Built with Next.js (React), TypeScript, Tailwind CSS, and Server-Sent Events (SSE) for real-time dashboard streaming without page reloads[cite: 11, 17].
Grounded AI Layer: Integrated Gemini / Groq / local Ollama strictly to summarise pre-verified anomaly JSON into plain English, backed by deterministic Jinja2 templates if the LLM API fails or exceeds 4 seconds[cite: 18, 24].
Challenges we ran into
- Raw-Byte Signature Verification vs. JSON Parsing: PayPal's
verify-webhook-signatureendpoint hashes the exact raw HTTP body bytes[cite: 6, 7, 22]. Standard framework JSON parsing alters whitespace and key ordering, breaking signature checks[cite: 34]. We solved this by capturing and persisting the rawBYTEAstream before deserialising any JSON[cite: 21, 22, 34]. - Transaction Search Latency: Because PayPal's Transaction Search API can lag real time by up to ~3 hours[cite: 4, 6, 7], relying on it alone would trigger false unknown-payment alerts (
A8)[cite: 35]. We combined fast per-order polling against Orders v2 (15-second demo interval) with a delayed 31-day Transaction Search sweep that ignores transactions newer than a multi-hour safety margin[cite: 4, 7, 13, 35]. - Guaranteeing Zero Double Fulfilments: A delayed webhook can arrive seconds after a background poll has already healed an order[cite: 29]. Enforcing relational
UNIQUEconstraints onevent_idandidempotency_keyat the database level—alongside stale-state filtering in our state machine—ensured retries and late arrivals remain strict no-ops[cite: 20, 21, 29]. - Preventing AI Hallucinations on Financial Records: Letting an LLM decide whether money moved creates unacceptable financial risk[cite: 4, 10]. We isolated AI strictly to explanations: deterministic rules handle all detection and healing, while the LLM receives only structured JSON facts, undergoes output validation for order ID, amount, and anomaly type, and falls back to Jinja2 templates[cite: 10, 24].
Accomplishments that we're proud of
- End-to-End Deterministic Core: Built a working state machine, reconciler, and idempotent Heal Engine that detects orphaned checkouts in under 60 seconds (with a 15-second poll target) and prevents duplicate fulfilments via database constraints and idempotency keys[cite: 4, 20, 21].
- Chaos Lab & 12-Scenario Resilience Matrix: Built an interactive fault proxy capable of injecting 4 distinct failure modes (
DROP,DUPLICATE,DELAY,REORDER) and validating system safety across 12 automated test scenarios with a target of 0 duplicate fulfilments[cite: 4, 16, 28, 29]. - Full Offline Mock Mode Parity: Recorded real PayPal API and webhook fixtures so the entire application runs deterministically without external network dependencies[cite: 11, 18, 32].
What we learned
- Deterministic First, AI Second: Keeping financial detection and healing 100% rule-based while using LLMs strictly for plain-language explanations gave us testable correctness alongside an intuitive UX for non-technical merchants[cite: 6, 10, 24].
- Idempotency Belongs in the Database Schema: Application-level checks alone are vulnerable to race conditions; encoding uniqueness directly into database constraints (
event_id,idempotency_key) makes double-processing structurally impossible[cite: 20, 21]. - Chaos Testing as a First-Class Feature: Building fault injection directly into the product accelerated debugging of race conditions and turned resilience from a claim into a repeatable, visual proof[cite: 4, 10, 14].
What's next for PAYPAL_AI
- E-Commerce Platform Plugins: Building native Shopify and WooCommerce adapters alongside Slack, email, and WhatsApp alerts so SMB merchants can onboard without writing custom code[cite: 14, 35].
- Multi-PSP Adapters: Extending the provider-agnostic state machine with adapters for additional payment gateways[cite: 14, 35].
- Automated Dispute Evidence & Finance Reconciliation: Turning stored order timelines into automated chargeback evidence PDFs via the PayPal Disputes API and adding payout-to-order reconciliation for finance teams[cite: 12, 35].
- Production Multi-Tenancy: Adding OAuth authentication, distributed queue workers, and per-merchant learned SLA thresholds[cite: 20, 25, 35].
Built With
- apscheduler
- docker
- fastapi
- gemini
- groq
- javascript
- jinja
- next.js
- ollama
- paypal
- playwright
- postgresql
- pydantic
- pytest
- python
- react
- sqlalchemy
- sqlite
- supabase
- tailwindcss
- typescript
- uvicorn
- vercel