Doorbell

Finds the PayPal webhooks your shop never heard or saved

Comment

Doorbell finds the payments, refunds and disputes that PayPal knows about and a shop's books do not. For each gap it says what it can prove and what it cannot. An AI points at the changed line of code that explains it, plain code tests the pointer by replaying real signed events, and PayPal is asked to resend only when it lists the event. It ends in a certificate that prints what it cannot prove first.

The scan, the stories and the certificate run on the demo shop in this repository. A shop of your own can install the reporter, so that a Doorbell you run yourself has your handler's own answers; what that gives and what it still lacks is in the README under "For a shop of your own".

Inspiration

When a webhook does not arrive, there is no error to look at. Other people have hit the same thing: a GitHub issue on PayPal's JavaScript SDK repository, titled "Low-Frequency Bug in PayPal REST API: Missing BILLING.SUBSCRIPTION.CANCELLED Events" (https://github.com/paypal/paypal-js/issues/578), reports that a cancelled-subscription event was sometimes not sent, leaving the integrator's database out of step with PayPal's records; a Stack Overflow question, "Paypal webhook when subscription ends/expires not fired" (https://stackoverflow.com/questions/72966268), asks about a subscription-ended webhook that did not fire.

I have been on the other side of this. At a shop I worked on, I once tightened the webhook handler's signature check. For a while after that, the payment provider's cancellation messages were rejected, and nothing logged an error. The books stayed wrong until someone compared them with the provider's records. The fix itself was small. What I could not answer was whether it was really fixed, and what else had been dropped in the meantime.

A tool can only see what PayPal tells it, so it cannot promise that nothing else was dropped. We made that limit part of the output. The certificate lists what it cannot prove before it lists what it checked, and a test fails if the product, or these documents, use a word from a list of words that overclaim.

What it does

There are six recorded stories, and each has the same seven chapters: what happened, how it travelled, why, the replay, the fix, ring again, the certificate (the recording of the resend that never arrived, made on 2026-10-19, has no patch block).

In the lead story a refund of $48.00 went through at PayPal and the shop's books still say paid: "PayPal refunded $48.00. Your books still say paid." "No error anywhere." Doorbell shows that the message reached the shop and was answered with a 200: "Each hop said OK. The books never changed." The AI points at the two changed lines of the shop's handler that answer first and save later, and two plain-code checks accept the pointer. A replay of the same real signed events through the last good build and the bad build shows the difference: "Same messages, two builds. Only one saves them." The AI also proposes a patch, shown as "Proposed by Claude" and "not deployed in this demo", after code checks and a replay with controls, including a tampered copy that must stay rejected. Then Ring again: because PayPal lists the event, Doorbell asks for a resend, sees it arrive, re-reads the books, and only then calls the row tied: "The books now say refunded." On the recorded pages this is a playback of a real resend, with its long waits cut. The certificate counts what was checked and prints the limits first.

The other stories are there to show what happens when it does not work out. A late payment overwrites a refund. A shop rejects PayPal's signed messages with a 401. A resend is accepted and never arrives, and the certificate stays red: "Order #2 now says refunded. Order #3 is still open." A setting, not the code, is wrong, the AI abstains, and the product says so: "The AI found no changed line that explains this gap." A dispute is listed by PayPal and the shop's webhook never asked for dispute events: "No webhook of this app asks for dispute events." A second page, the owner view, says the findings in plain sentences, with the two limits an owner needs.

Here is what is real and what is not. The six stories were run on the deployed shop against PayPal's Sandbox; the PayPal ids on screen are PayPal's, the money is Sandbox test money and the shop is fictional. Each bug is one we planted, and the page says so on each screen: "Sandbox demo · fictional shop · PayPal test money · bugs are planted by us". The recorded pages play back stored runs and call nothing: "Nothing is sent to PayPal." The live lane, with a judge token, runs a new order, a planted bug and a refund against the Sandbox and takes a few minutes. The dispute in its story was made by PayPal's Sandbox-only test call, not by a customer.

Doorbell says what it cannot prove. A resend can be accepted and never arrive: "Resend accepted, not delivered". The cause may not be in the code: "The cause is not proven." An event that is not in PayPal's event list cannot be shown to have existed. The certificate is not signed. The Sandbox is not production, and we quote no rate for how often this happens there.

How we built it

Next.js 16 and TypeScript on Vercel, with Upstash Redis for state, PayPal's Sandbox REST API and webhooks for the whole PayPal side, and Claude through the Anthropic API for the two model steps. The code was written with Claude Code; the owner read each patch proposal and each public text before it was used.

The project is about PayPal's webhooks end to end, so most of that API surface is in use: a second webhook for the witness and verify-webhook-signature on each delivery; the webhook management calls, to read which event types the shop's webhook subscribes to; the events list; the resend call; the Orders and payments calls (capture read, refund) for the demo shop, the refund through PayPal's Agent Toolkit; and the Disputes API, read only.

The facts are decided by code. A gap is a capture whose PayPal answer differs from the shop's books, found by asking PayPal again. In three scans of a 24-capture session in which the shop had dropped 12 refunds, Doorbell found 12 of 12 in each of 3 scans, with 0 false alarms in each, and each scan took 2.8 to 3.2 s of server time. In 9 of 9 trials of the demo shop's three planted bugs the evidence class was the expected one.

The AI proposes and code checks. The cause call returns one of eight categories and the changed lines it cites, and nothing else; the sentence on the card is a template. The cited line must be a changed code line, not a comment, and the category must fit what the shop answered. The replay then confirmed the cause in 9 of 9 replays, on those same three planted bugs; a plain revert fixes all three, so the demo does not show the model doing better than a revert. Input goes through one normalization (invisible characters, look-alike letters, secret-shaped strings, delimiters), the prompt tells the model the diff is untrusted data, any failure ends as "Not diagnosed.", and a live call has to pass an hourly limit and a money budget per day. 3 of 3 recorded patch proposals, one per planted bug, passed the checks and the replay with controls; the patches are never deployed, and a person read each one before it was allowed on a screen.

The hosted default is a set of recordings: each recorded run is a list of snapshots captured from the deployed shop against the Sandbox, imported as JSON, and the recorded pages make no network or database call. The live lane renders the same snapshot shape from a real session. Before the AI step was built, we measured how the Sandbox behaves and wrote it down; the numbers in this text come from those measurement files through a generated ledger in the repository's judge pack, and a test fails when a figure and its row differ.

Challenges we ran into

  • A resend is not a delivery. PayPal accepts a resend with a 202 and sometimes nothing arrives. A resend of an event the shop had rejected three times reached the shop in 4 of 37 sends, against 12 of 22 when the shop had answered 200. So Doorbell resends only when the event is in PayPal's events list, looks for the arrival, and ends one story on "Order #3: Resend accepted, not delivered. The gap stays open."
  • Absence is hard to claim. PayPal's events list lags, so "not listed" proves nothing at first. A claim that an event is missing waits six minutes; in the test of that rule there were 0 claims before the window.
  • Replay needs old signatures to verify. We could not assume PayPal still accepts a stored signature: 266 of 266 stored real signatures were still accepted, the oldest 25.25 h old.
  • The model must never decide a fact. Whether there is a gap, whether a row is tied and the colour of the stamp are code. Instructions planted in the code did not move the model: 8 of 8 injection cases were right in all runs.
  • Not overclaiming. The certificate's green only ever says "Tied on 3/3 of what we can see." We keep a list of words the product may not use, and a test scans the product and these documents for them.
  • Cost and abuse. A flood of 1000 scan requests let 8 of 1000 through, and 500 further blocked requests used 0 commands of the database. One target was missed: 10 minutes of polling used 102 vs 60 database commands, and in a replay of the page's own polling a page left alone used 49 vs 60 and a page in use 266 vs 60, so we restated the target on 2026-10-21.

Accomplishments that we're proud of

  • The chain from the shop's books to PayPal's own answer, to the changed line, to a replay of real signed events, to a resend that is attempted only when it can arrive, to a certificate that lists what it could not prove, working end to end on the Sandbox, live and as recordings.
  • A product that can end on an open gap. One recorded story ends with a red certificate and says why.
  • A control that catches a bad fix: a patch that removes the signature check passes a plain replay, because the shop then books any event it is sent; the tampered-copy control rejects it, and the screen shows that.
  • Each measured figure that comes from a measurement file is a row of the numbers ledger; the few that come from the build log are marked as such in the judge pack. The sealed evaluation sets were committed by hash as the first commit of the public repository, before any model was run on them.
  • Nobody needs a key or an account to see it: the hosted site plays back real runs and says so.

What we learned

  • In 3 of 3 trials the Sandbox's retries stopped after 3 tries, and a shop that answered 200 gets no retry at all. PayPal's own events list lags, and a resend that PayPal accepts may not arrive. A tool that asks for a resend has to see the arrival, not trust the 202.
  • Disputes: PayPal's Disputes API listed a dispute made by the Sandbox test call 39 to 104 s after it was made. A webhook hears only the event types it subscribes to: with no subscribing webhook there were 0 events in 483 s of watching in the one test, and with a subscribing webhook 4 created and 4 updated events were listed for 4 disputes. So a shop can be asked for $48 back and never be told, and Doorbell finds it by asking PayPal.
  • Where a plain revert of the bad deploy is enough, the model adds nothing; an offline spike said so (judge pack, section 4). On the development set a plain-code baseline got the category right in 11 of 19 cases and the model in 19 of 19 cases, but that set is the one the prompt was built on. On the held-out sets, sealed by hash before any run and evaluated on 2026-10-26, the model named the right category in 34 of 35 cases written in a separate session (the regular-expression baseline 15 of 35, a sealed checklist baseline 21 of 35) and in 11.67 of 12 cases rebuilt from real public fix commits; an obfuscated twin of those commits scored 12 of 12, so nothing suggests the model knew them. One answer was confidently wrong. The labels were audited by a separate Claude agent, not by a person; one ambiguous label was excluded before the run.
  • Most of the honesty work turned out to be layout and tests: the certificate's sentences are templates with numbers filled in, and the banned-word list and the quoted sentences are checked by tests.

What's next for Doorbell

  • Widen the held-out evaluation: the real-commit set covers 3 of the 8 cause categories and its labels were audited by a separate Claude agent, not by a person.
  • Pointing the scan at a shop's own orders and books. The reporter for a shop of your own is built: a signed HTTP reporter for the shop's handler and an intake that a Doorbell you run accepts only with the shared secret, so the classes that depend on the shop's own answers can be told apart outside the demo shop. Reading the shop's books is not built, so the gap list is still the demo shop's.
  • A read-only audit of your own Sandbox app. Not built yet.
  • Production. The demo is Sandbox only, and the behaviour of PayPal's production retries is documented, not measured by us.

Built With

Share this project:

Updates

Submission history