Sentrix: AI risk checks for PayPal payments

Sentrix puts a risk gate between PayPal authorize and capture: rules, a model and an AI analyst judge each payment, agents pay only inside written mandates, and a person decides the doubtful ones.

Comment

Inspiration

PayPal splits a card-style payment into authorize and capture. Between the two there is a window where the money is held and nothing has been taken, and that is where a payment can still be judged. Most fraud tools either block too much or wave things through, and when an AI says "suspicious" nobody can see why. At the same time shopping agents are starting to pay on people's behalf, and a human who hands an agent a card has no written limit and no way to see what it did. We wanted one system where the decision is made in that window, every part of it can be read afterwards, and an agent can only spend what its owner wrote down.

What it does

A payment goes through six steps, and each one is on record.

  1. PayPal checkout. The buyer approves in PayPal's own window. PayPal authorizes the money; nothing is captured yet.
  2. Risk rules. Amount against the buyer's usual spend, payment speed and a new or missing device add up a score. An admin edits the rules; 70 or more blocks, 30 or more holds.
  3. Model score. A small logistic regression gives a second score with its top reasons. It can ask for a hold, never a block.
  4. AI analyst. Gemini reads a facts-only payload (bands, flags, rule names, no ids, emails, IPs or device ids) and returns approve, review or block with an explanation. It can escalate a payment to a person but can never void or release one. Every text it writes is checked: a number, rule name or identifier that is not in the facts replaces the AI text with a fixed one built from the facts.
  5. Human gate. A reviewer sees the rules, the model and the analyst side by side and decides a held payment within a decision limit. Above the limit, or for a refund, an admin settles it (two people).
  6. Capture or void. Sentrix makes the PayPal call. Capture charges the buyer, void releases the hold. Idempotency keys make a repeated click safe.

Around that core:

  • Agent mandates. A customer gives a shopping agent a written permission: one merchant, a cap per payment, a cap in total, an end date, revocable at any moment. The agent presents a secret token (stored only as a digest). A payment outside the mandate is refused and audited; a payment inside it is still judged by the rules, the model and the analyst. A reviewer sees "Paid by an agent" with the mandate on the case and can revoke it there. The owner gets a notice per agent payment.
  • Reviewer decisions are labels. A person's capture or void of a held payment becomes a label. The Feedback page scores the rules, the model and the analyst against those labels and shows each rule's false alarms.
  • Rule changes with a replay. When a person overrides the rules, an agent drafts one change to one rule and replays it over every stored payment (false alarms, misses and held counts before and after). It never edits a live rule: only an admin accepting the draft does.
  • Reading a case. "Why this verdict?" and "What would change it?" answer from the same facts, "Similar past cases" shows the nearest decided cases (pgvector) with what a person decided, "Linked accounts" shows accounts that share a device, and "Draft case narrative" writes a regulator-style summary saved to the audit trail.
  • The customer side. A held customer sees what the hold means and can send the reviewer one note.
  • Operations. Incident digest (grouped failures with a drafted GitHub issue an admin approves), deep health, JSON logs with request ids, rate limits, a circuit breaker around the AI call, a signature-verified PayPal webhook receiver with event-id dedupe.

How we built it

  • Backend: Rails 8 API, PostgreSQL with pgvector, Solid Queue and Solid Cache (no Redis). PayPal Orders v2: create, authorize, capture, void, refund, and webhook signature verification. A SimulatedPaypalClient runs the same decision path for demo traffic and for agents, which cannot click through a PayPal approval window.
  • AI: Gemini (gemini-3-flash-preview, free tier) behind a token bucket, a circuit breaker and one capped retry. Strict JSON output, a timeout and a fallback to rules only. The analyst's text is checked mechanically against the facts it was given.
  • Frontend: React, TypeScript, Vite and Tailwind. Navigation and landing page per role (customer, reviewer, admin), tested at 1440 px and 375 px.
  • Model: the second score was trained on Kaggle. A retrain on reviewer labels was compared with the shipped model on a fixed hold-out and was not shipped, because it did not clear the bar we set before running it.
  • Free stack: Vercel (frontend), Render (API), Neon (Postgres), Gemini free tier. Sandbox money only.
  • Process: every feature was a branch, a PR with tests, green CI and CodeQL, then a live check on production before the next one.

Challenges

  • The PayPal approval window cannot be driven by an agent, so agent payments use the simulated client. The decision path is real; no money moves. We say so everywhere it matters.
  • Free-tier AI models that answer slowly or with a 503. The analyst has an 8 second timeout. On a structured request one lighter model took about 15 seconds and failed that timeout every time; we timed several models and run the one that answered in about 3 seconds. The analyst sits behind a breaker with a rules-only fallback, so a failed call never blocks or releases a payment.
  • An AI that sounds sure. A confident paragraph with an invented number is worse than no paragraph. The grounding check replaces such text instead of showing it.
  • Honest numbers. The retrained model did not beat the shipped one, the similar-cases check came out equal to the baseline, and the demo labels are scripted. We recorded each of those instead of rounding them up.

Accomplishments we are proud of

  • A payment decision a person can follow end to end: what fired, what the model and analyst said, who decided, what PayPal was told. Every step is in the audit log.
  • Agents that can only spend inside a written mandate, with refusals audited and the owner told.
  • An AI that is bounded by construction: facts-only input, can escalate but never release, and every sentence it writes is checked against the facts. The checker accepted 10 of 10 faithful texts and rejected 12 of 12 planted fabrications in its test corpus (a synthetic corpus, not a rate for the live model).
  • Measured, not asserted: on the 30-case offline evaluation the rules alone agree with the label in 20 of 30 cases and the AI analyst in 27 of 30 (measured on the model we evaluated, gemini-3.1-flash-lite; the live demo now runs gemini-3-flash-preview, which that evaluation did not cover); on a load test at 0.5 CPU and 512 MB the list and staff paths stay flat up to 40 requests a second with 0 server errors.

What we learned

  • The useful place for an LLM in a payment flow is next to the decision, not in it: it explains, escalates and drafts, and a person or a rule keeps the power to act.
  • Put the limit on what the agent may ask for (the mandate) and still judge every payment inside it.
  • A label you get for free from a reviewer's click is worth more than another model: it scored the rules, showed their false alarms and drove the rule-change drafts.

What is next

  • An operations view on the Agents page: each mandate's payments with their outcome and the refusals that hit it.
  • A real signed PayPal webhook delivery recorded end to end (the receiver is tested; production has not yet shown a signed delivery).
  • Real reviewer volume: the demo decisions are scripted, so the agreement figures are on synthetic data until people use it.

Significantly updated after 2026-10-01

Sentrix began as fintech-fraud-detection, a transaction risk-scoring app (rules, device fingerprinting, background jobs, a React frontend), built before the hackathon period together with a collaborator who wrote its original frontend. That earlier work is disclosed as pre-existing and is not claimed here. At the start of the period the repository had 11 commits and no PayPal code in the backend. The update below was built and merged by the submitter, solo, between 2026-10-21 and 2026-11-03; the pull requests are in the public repository.

  • PayPal: Sandbox orders: create, authorize, capture, void, refund behind the risk gate; payment lifecycle (hold expiry, partial refunds, insufficient balance); idempotency keys; signature-verified webhook receiver with event-id dedupe. Pull requests: #33, #44, #45, #42, #81.
  • AI: Bounded analyst verdict, evidence timeline, resilience (token bucket, breaker, retry), ask the analyst, case narrative, grounding check on every AI text. Pull requests: #35, #40, #43, #74, #75, #79, #87, #90.
  • Scoring: Explainable second score; editable risk rules; reviewer decisions as labels; Feedback page with per-rule false alarms; rule-change drafts with a replay over stored payments; retrain comparison. Pull requests: #30, #32, #49, #80, #64, #65, #54.
  • Agentic commerce: Agent mandates (scope, caps, expiry, revoke), agent context on a case, owner notices, demo agent persona. Pull requests: #86, #89, #91.
  • Review and roles: Review queue and agent actions with admin limits; analyst, reviewer and admin roles with decision limits; four-eyes approval for over-limit decisions and refunds; customer status and note on a held payment. Pull requests: #36, #37, #46, #47, #88.
  • Case reading: Account and device ring graph; similar past cases (pgvector). Pull requests: #73, #82.
  • Product and UX: Role-based app shell, Home with live pipeline counts, sign-in and register redesign with sandbox logins, Insights, My Spending, Help, accessibility and 375 px pass, USD formatting. Pull requests: #50 to #63, #76, #77.
  • Operations: Rate limits, deep health, JSON logs with request ids, warm-up ping, indexes and keyset pages, incident digest, load test with measured numbers, demo traffic generator. Pull requests: #38, #39, #66, #69, #70, #71, #72, #48, #60.

After the main build, on 2026-11-03: MIT license (#94), dark README screenshots and a note on the production AI model (#95), the demo video link (#96).

Built With

Share this project:

Updates

Submission history