Inspiration
Subscription businesses lose revenue every month to failed renewals. Most failures are soft declines, like insufficient funds or a temporary bank issue, and a better-timed retry can win them back. But merchants rarely know how much they're losing, and they have no fair way to test whether a new recovery tool works on their own customers before committing. For PayPal, that stalls the move from free trial to enterprise contract. We wanted to replace "trust us" with a controlled experiment the merchant runs themselves.
What it does
SecondMint is a measurement layer for payment recovery, built for subscription merchants and the PayPal teams that sell to them.
- Connect: syncs the merchant's Braintree customers, subscriptions and every payment attempt.
- Revenue Leak Diagnostic: shows which renewals fail, why, how many the current process recovers, and the dollars still leaking.
- Recovery trial: splits customers at random into two routes. Route A keeps the merchant's current retry process; Route B uses PayPal's Smart Retries. A 35-day trial is fast-forwarded, and every retry is a real Braintree sandbox charge, with no double charges.
- Results report: fixed statistical rules give the verdict: uplift in percentage points, a 95% confidence interval and net dollars recovered. If the data can't support a verdict, it says 'Inconclusive' and shows how much volume a conclusive test would need. Exports as a one-page PDF.
- AI analyst: the merchant asks questions in plain English (Which decline codes drove the result?). Answers cite the trial's own figures, and code rejects any answer containing a number that isn't in a tool result, so it can't invent numbers.
- PayPal wallet experiment: tests whether adding a PayPal button to the payment-update page recovers more customers who have to act. A customer can save their PayPal wallet without paying (Vault setup-token flow), and the next renewal is charged to it with no customer present.
Our demo results (10,000 sandbox customers, split 50/50): | Recovery trial | Current process (Route A) | PayPal route (Route B) | |---|---|---| | Failed renewals | 1,035 | 1,021 | | Recovered | 285 | 397 | | Recovery rate | 27.5% | 38.9% | | Net revenue recovered | $22,045 | $30,073 |
Verdict: PayPal route wins. Uplift +11.3 pp (95% CI +7.3 to +15.4, p < 0.0001), worth about $16,756 in incremental net revenue across the trial's failed invoices. Fixed rules decided the verdict, not the AI. To be transparent, this run was a favorable draw: across 20 simulated runs the average uplift was +6.8 pp, close to the +7 pp benchmark we assumed.
| Wallet experiment | Result |
|---|---|
| Card-only conversion | 24.8% |
| Card + PayPal conversion | 25.3% |
| Uplift | +0.5 pp (95% CI −2.8 to +3.9) |
| Verdict | Inconclusive |
SecondMint warned before the wallet test started that the expected +2.3 pp lift was smaller than the sample could reliably detect. It doesn't turn every positive number into a win.
How we built it
- Next.js + TypeScript web app
- Braintree sandbox (PayPal): customers, vaulted cards, subscriptions, every renewal and retry, with custom fields carrying the billing history
- PayPal REST APIs: Checkout and Vault (setup tokens) for the wallet experiment
- AI (DeepSeek
deepseek-chatvia an OpenAI-compatible API, Gemini free tier as fallback): writes the Leak Diagnostic summary and the Results summary (which can't contradict the rule-based verdict), and powers the AI analyst, which answers by calling tools over the same data the screens use. Every number in an AI answer is checked against the data; answers that fail are retried or replaced by a template. - APIMatic: every PayPal REST call in the wallet flow goes through PayPal's official APIMatic-generated server SDK (
@paypal/paypal-server-sdk): Orders (create, capture) and Vault (setup token, payment token, saved wallets), with typed models and SDK-managed OAuth. - AG Grid: interactive tables for failed renewals (sort, filter, group, drill-down), the results breakdown and the wallet results.
- Recharts: leak and timeline charts.
- A seed script creates a realistic SaaS merchant: 3 plans, 10,000 customers and about 90 days of failed renewals, retries, recoveries and refunds (about 44,000 transactions)
- Trial runs live on the server with a ledger, so pause, resume and crash recovery never charge an invoice twice
- 109 automated tests and a WCAG 2 AA accessibility scan
Challenges we ran into
- A sandbox can't act like a real bank. It can't tell us how a bank would answer a retry, and Smart Retries can't be switched on per customer in a sandbox. So retry outcomes come from market-benchmark assumptions, each visible and editable in the app with its real-world source cited. We never tune them to force a win.
- Small effects need big samples. The wallet experiment's expected lift (+2.3 pp) was too small to detect even with 10,000 customers, so we built a pre-test power warning and kept the honest "Inconclusive" result rather than inflating the assumptions.
- Sandbox throttling. Seeding and running tens of thousands of real sandbox charges meant tuning concurrency and backoff, without ever creating a duplicate charge.
Accomplishments that we're proud of
- Every customer, card, subscription and payment attempt is real sandbox data. Only the bank's answer to a retry is modeled.
- The report is honest by design: it says what's real, what's assumed, and when the data isn't enough.
- The AI analyst is grounded: adversarial prompts are refused, not answered with made-up figures.
What we learned
In a payments experiment, 'trust is the product'. A merchant acts on a result only if they can see how it was measured. We also learned how much recovery comes down to retry 'timing', especially retrying insufficient-funds declines around paydays, and how much volume it takes to prove a small but valuable uplift.
What's next for SecondMint
In production, Route B becomes a Braintree merchant account with Smart Retries 'on', and Route A keeps the merchant's current process. Real banks decide every retry, and the same assignment, ledger, statistics and report measure the real uplift. Next steps: a pilot with a real merchant, and packaging SecondMint as a proof-of-value tool for PayPal's enterprise sales team.