Hard Cap

Inspiration

Every AI product eventually promises a customer a hard spending cap, and almost every implementation of that promise is a lie under load.

The reason is specific. You do not know what a request costs until after the model has answered. So the obvious code — read the balance, decide, then charge what it actually cost — has a window between the decision and the charge. Under concurrency, a hundred requests can all read the same healthy balance and all decide yes. Add at-least-once queues that duplicate settles, workers that crash holding a claim, and stalled workers that wake up minutes later and try to settle against a hold that was already reaped, and "hard cap" becomes a preference.

We wanted the cap to be a property of the data structure rather than a promise made by the calling code.

What it does

A per-tenant hard budget cap for inference spend, built on reservations rather than charges, and an adversary that tries to break it.

• RESERVE places a hold for the estimated maximum cost before the model is called. SETTLE replaces that hold with the true cost afterwards. RELEASE cancels it. EXPIRE reaps holds from workers that died. Available balance is cap - settled - outstanding_holds, so the money is committed before it is spent, which is what closes the window.
• Admission is atomic via a compare-and-swap loop over a versioned balance record. You can watch the retries happen live.
• Fencing tokens mean a worker that stalls and wakes up late cannot settle against a hold that has already expired and been reissued to someone else.
• Idempotency keys make duplicate delivery a no-op.
• The ledger is append-only, double-entry, and hash-chained with SHA-256, plus an independent replay auditor that reconciles entries to the balance.
• A naive mode swaps in the obvious check-then-charge implementation and runs the identical adversary with the identical seed, so the contrast is measured rather than argued.

How we built it

Vanilla JavaScript, Web Workers, typed arrays, and a hand-written zero-allocation SHA-256. Ledger records are 48 bytes in a preallocated typed array.

Concurrency comes from a seeded cooperative interleaver: a generator-based green-thread scheduler with explicit preemption points inside the critical region. That is a deliberate choice. It makes every run reproducible from a seed, and it makes "atomic region" a precise, inspectable thing rather than a hope about the JS event loop. The Web Worker exists to keep the UI responsive, not to provide the parallelism.

The adversary generates a deterministic, position-addressable job stream with fault injection and tracks 13 named race types.

Challenges we ran into

The most useful bug was in the adversary, not the engine. periodCloseReap — the race where a billing period closes while reservations are still in flight — fired exactly once across three million operations. That looked like a rare race. It was actually a dead test: every tenant exhausted its budget early, so the entire tail of each period was denial-only with nothing in flight to reap.

Fixing it meant capping billing periods to a fixed traffic slice and giving tenants heterogeneous oversubscription factors so some still hold live reservations at close. It now fires 1,442 times. A race you never trigger proves nothing, and it took a suspiciously round number to notice.

Accomplishments we're proud of

Identical adversary, identical seed, 3,000,000 operations at concurrency 64:

SAFE: 0 violations, chain valid, cap never crossed. Peak settled $34.32 against a $36.96 cap, with max per-tenant utilization of exactly 100.00% — it rides the line without going over, which is the actual hard part. 6,127,144 CAS attempts, 4,492,944 retries, 2,248,022 ledger entries, 379,182 ops/sec, 12.07s.

NAIVE: breaches in 199 of 199 billing periods, 88,286 violations, cumulative overspend $392.35, peak 115.59% of cap.

Every race type fired, none zero: casRetry 4,492,944 · admissionDenied 511,386 · expiryVsSettleRace 284,771 · staleFenceVsReissuedHold 347,539 · lateSettleExpired 490,206 · doubleSettleAttempt 14,120 · settleAfterRelease 11,568 · duplicateInFlight 39,274 · duplicateReplay 132,916 · crashRecovery 40,860 · clockSkewIgnored 42,165 · costOverageClamped 7,743 · periodCloseReap 1,442.

Tamper detection: altering one historical amount produces CHAIN BROKEN — entry #831,768 amount altered, chain diverges at #831,768. A clean walk of 2.2M entries takes 3,957 ms with 0 audit violations.

Honest limits

The concurrency is simulated. A seeded cooperative interleaver is not OS threads. It is the right tool for proving an invariant reproducibly, and it is not evidence about multi-core behaviour.

Naive mode is the strongest reasonable naive build — its charge is atomic, its admission uses a worst-case estimate, and it consumes the same job stream. The only thing it lacks is the reservation. A sloppier implementation would breach far worse, so $392.35 is a floor, not a worst case.

The naive per-period overshoot is 15.6%; the $392.35 headline is cumulative across 199 periods (5.33% of authorized budget). We label it as a sum rather than implying one dramatic blowout.

verifyChain() on 2.2M entries blocks the worker for about 4 seconds with only a status label. Only the ledger lives in the worker — a page reload discards everything, as there is no storage layer yet. And the dollar figures are small because caps are calibrated from a measured pilot rather than set to a round marketing number.

What we learned

An invariant is worth exactly as much as the adversary that failed to break it. Most of the engineering value here was not in the reservation protocol, which is well understood, but in building fault injection aggressive enough that each of thirteen races fires tens of thousands of times — and then noticing when one of them quietly did not.

What's next

Persist the ledger to durable storage, replace the cooperative scheduler with real workers over SharedArrayBuffer and Atomics to test true parallelism, add a Postgres-backed reference implementation, and expose the whole thing as a small library with the adversary as its test suite.

Built With

• concurrency
• distributed-systems
• fintech
• javascript
• ledger
• property-testing
• saas
• sha-256
• typed-arrays
• web-workers