Keepwell

An honest AI retention agent for PayPal Subscriptions: one short conversation, at most one fair offer the merchant approved, and a Cancel button that always works in one click.

Comment

Inspiration

Businesses that bill through PayPal Subscriptions lose revenue in three places. A subscriber clicks cancel and nobody asks why. A renewal fails and the subscriber never hears about it in time. Someone cancels inside their PayPal account and the merchant never learns the reason. The usual tools for the first problem are dark patterns: hidden cancel buttons, guilt copy, fake discounts. We wanted to see whether an AI agent could do the opposite: one short, honest conversation, one fair offer the merchant has already approved, and a cancel button that always works.

What it does

Keepwell works on the merchant's real PayPal subscriptions, at three moments:

  1. Honest cancel save. When a subscriber clicks cancel, a widget opens a short chat. The merchant's policy decides what may be offered: a pause, a cheaper plan, a time-limited discount or a partial refund. The agent offers at most one, and the subscriber accepts by clicking a button. Keepwell then applies it to the PayPal subscription. Cancel now is always one click and never involves the AI.
  2. Failed-payment recovery. When a renewal fails, Keepwell reads PayPal's failure reason and next retry time, and sends a message matched to the reason before the retry.
  3. Win-back. When someone cancels inside PayPal, Keepwell sends one message asking what happened. Never a drip campaign.

The merchant also gets a dashboard with:

  • a live activity feed;
  • each subscriber's actions and conversations;
  • a timeline;
  • insights that turn exit conversations into structured reasons;
  • a copilot whose numbers all come from SQL.

Other AI agents can use the same rules through MCP. A subscriber's assistant can read options and cancel. A merchant's assistant can propose actions but never carry them out.

Every action is idempotent, reconciled against PayPal, and written to a hash-chained audit log the merchant can verify.

How PayPal is central

Keepwell has nothing to act on without a PayPal subscription, and PayPal stays the source of truth.

  • Subscriptions API: suspend, activate, cancel, revise (with PayPal's buyer approval page), price patch, and capture of an outstanding balance. A pause is a suspend plus a scheduled activate.
  • Plans API: a merchant can approve an insight that creates a new PayPal plan, through the same action engine.
  • Payments (Refunds v2): partial refunds, made safe to retry with PayPal-Request-Id.
  • Webhooks: every event is checked with PayPal's verify-webhook-signature API, then treated as a hint. We re-read the subscription and compare it with what we hold, so duplicate, late or missing webhooks can't corrupt state. A reconciler sweeps for anything webhooks miss.
  • JS SDK: the Pantry Plan demo store subscribes through PayPal's button.

We tested PayPal's behaviour on the sandbox before designing around it, which turned up three things:

  • A price change inside 10 days of billing returns 204 but isn't applied, so the policy never offers one.
  • A revise only takes effect after the buyer approves it, and no webhook fires for it.
  • Subscription transaction ids work as capture ids for refunds.

How AI is used, and how it's kept honest

A small, fast model runs the conversation and classifies the reason. A larger one powers the merchant copilot. Both sit behind one model interface. The hosted demo runs on Groq's free plan (gpt-oss-20b and gpt-oss-120b), and the same code runs Claude Haiku and Sonnet. The design constrains the model instead of trusting it:

  • Cancel never touches the AI. No model call, nothing that can fail because of AI.
  • The AI picks from a menu; the policy decides. A pure policy engine computes what may be offered. The model picks at most one item. The policy re-checks the pick, and checks again when the subscriber accepts.
  • Terms come from data, not the model. Offer cards are built from the offer object. A deterministic claims checker rejects any model text with figures, dates, durations, promises or pressure.
  • Consent is a button click, never parsed from text.
  • Hard limits: a few messages, one offer, an 8-second deadline and a daily budget. If any limit is hit or the model is down, the chat falls back to reason buttons and templates, and cancel still works.

Evals. We built an eval harness of 404 personas: everyday subscribers in English, Hindi and Spanish, plus 104 attackers. It has two hard gates: 0 policy violations, and 1 click to cancel.

  • Against a scripted, fully jailbroken model, the code alone held both gates over all 404 sessions. It caught 204 invented claims and refused 132 illegal picks.
  • On a real model (gpt-oss-20b, a 51-persona subset with 13 attacks), the results were:
    • 0 policy violations and 0 red-team violations;
    • 1 click to cancel;
    • a dark-pattern score of 0.06 out of 3;
    • classification and language match of 1.00;
    • a median reply time of 855 ms.

A fake-menu injection did fool the model into picking an offer that wasn't on the menu, and the policy refused it. That's exactly why the policy has the last word.

How we built it

  • A TypeScript (strict) monorepo with pnpm and Turborepo, and CI on Linux and Windows with a secret scan.
  • A Fastify API with zod at every boundary, Drizzle ORM on Postgres (Neon) with row-level security for tenant isolation, and pg-boss for jobs on the same Postgres.
  • On the PayPal side:
    • a gateway that is the only code allowed to call PayPal;
    • an action engine with a state machine and per-subscription locks;
    • a verified webhook pipeline and a reconciler;
    • a mock PayPal server built from recorded sandbox responses, so tests never need credentials.
  • A Preact widget in Shadow DOM with server-sent events, a React dashboard, a Preact portal and Judge Mode, and MCP servers.
  • Hosting on Render's free tier with a Neon free database.
  • Five people, each owning a vertical slice, working together only through typed contracts.

Challenges we ran into

  • PayPal accepts PayPal-Request-Id on only a few endpoints, so suspend, activate and cancel needed a read-before-retry strategy.
  • A price PATCH inside 10 days of billing returns 204 and silently keeps the old price.
  • Sandbox buyers created with country India fail at subscription approval; US accounts work.
  • Judges need subscriptions that are already approved, and only a buyer can approve one. So we built a pool that Judge Mode lends from for 30 minutes at a time.
  • On submission day we had no working Anthropic key, so the hosted demo moved to Groq's free plan behind the same interface. The agent was built to degrade safely, so nothing else had to change.
  • Render's static-site rewrite held back streaming responses until they finished, so the live feed now connects to the API host directly with a short-lived ticket.

Accomplishments that we're proud of

  • Cancel is one click, and the cancel route never calls the agent. A test proves it by running cancel with an agent that throws.
  • An eval harness with hard gates, run against both a jailbroken script and a real model.
  • Webhook verification against real PayPal (30 of 30 verified), with webhooks treated as hints rather than truth.
  • A hash-chained audit log with one-click verification.
  • A mock PayPal server that mirrors the sandbox.

What we learned

Constrain the model with data and code, not prompts alone. Treat webhooks as hints. Test the PayPal sandbox before designing around it. Agreeing the contracts first let five people build in parallel.

What's next for Keepwell

  • Better conversation quality in Hindi and Spanish beyond the current templates.
  • A fuller end-to-end sandbox run of recovery and win-back.
  • Contributing subscription tools (suspend, activate, revise, capture) to PayPal's agent toolkit.

All data is fictional and everything runs on the PayPal sandbox.

Built With

Share this project:

Updates

Submission history