The claim

Tailrace is a payout pipeline you can kill halfway through sending money, and it will still pay everybody exactly once.

Each stage runs as its own durable Render Workflows task. There is a button that crashes the first payout step deliberately, after PayPal has already accepted the batch, which is the exact window where double payment happens. The retry adopts the existing batch instead of creating a second one.

When an item fails, an agent proposes a fix and a rules gate approves or rejects it. It can suggest a correction; it can never move money alone.

The expensive bug

Here is the sequence that costs real money.

Your job sends a batch of payouts. PayPal accepts it and returns 201. Then your process dies — a deploy, an out-of-memory kill, a dropped connection. The job restarts, sees no record of having finished, and sends the batch again.

Now you have paid twice.

This is not exotic. It is the ordinary failure mode of any long-running job that moves money, and the window is small but entirely real: after the external system has accepted the work, before your own system has written down that it did.

The usual advice is "make it idempotent," which is correct and skips the hard part. Idempotency has to hold at the exact moment things break, and the only way to know it does is to break them.

So there is a button that breaks it

Queued

Load six supplier payouts, then press Send, then crash mid-flight.

The pipeline splits them by currency, sends a GBP batch and a USD batch in parallel, and then the first step is killed deliberately — after PayPal has accepted the batch. Exactly in the window where double payment happens.

The retry

The retry recognises its own work. PayPal's own response, visible in the run:

retries=1 · 3 items, batch VZGL2JXLMDMG8, PayPal already had it

The batch was adopted, not recreated. Six payouts went out, six were accounted for, none twice.

Every step is its own task

The pipeline

Each stage runs as a separate Render Workflows task, chained together. Reading the queue, submitting a batch, waiting for a terminal state, reconciling against PayPal — each one can fail and retry on its own without dragging the rest down with it.

That is why the crash test works. A pipeline that is one long function cannot be killed in the middle and resumed sensibly. One made of durable tasks can.

A 201 from PayPal is treated as the start of the story, not the end. The run polls each item until it reaches a final state: landed, held by PayPal as unclaimed, failed, or escalated.

What it cannot fix, it hands over

When a person is needed

When an item fails, the agent reads the payee record and the PayPal error code and proposes a fix. A fixed rules gate approves or rejects every proposal, so the agent can suggest a correction but cannot move money on its own.

And where no safe fix exists — a recipient address that is wrong in a way only a person can confirm — it escalates instead of guessing. One payout in the demo ends up there, and that is the correct outcome rather than a failure of the agent.

Webhooks

PayPal sends several event types per batch, each carrying the same re-read line. They are grouped so a batch reads as one row rather than six near-identical cards.


Pipeline	Render Workflows — every stage its own durable task
Database	Render Postgres — the ledger, read back rather than trusted from memory
Agent	Claude Sonnet 4.5 via the Anthropic API, behind a rules gate
PayPal	Payouts v1 (idempotent on `sender_batch_id`), Invoicing v2 for chasing overdue invoices, signed webhooks

Two things worth knowing if you build on Render

Workflows have a free tier. Our blueprint asked for plan: starter and got a 402 demanding a card. The type was never the problem — the plan was.

The workflow has to start through the render-workflows CLI. Running python workflow_main.py fails with ValueError: RENDER_SDK_MODE environment variable is required. The CLI sets that variable and the socket path; the platform injects them only into genuine Workflow services. Creating a service through the REST API with type: workflow silently produces a web_service instead, which is why the SDK kept failing until we deployed through a Blueprint.