Inspiration

Invoice fraud is boring, and that's why it works. A "vendor" sends a perfectly normal bill with one small change: "we've switched banks, please pay this new account." People fall for it every day. Now imagine an AI agent paying your bills for you. It reads the PDF, trusts the PDF, and sends the money. We wanted to build a payment agent that's genuinely useful but that you can't talk into paying the wrong person.

What it does

Poisoned Invoice reads freelancer invoices and puts together a monthly PayPal Payouts run (sandbox only). The core rule is simple: the AI never decides who gets paid. The payee always comes from the owner's own vendor list, never from the invoice.

If an invoice asks to change bank details or contact info, it's held, and the change has to be confirmed on the phone number already on file before any money goes anywhere new. Plain, testable rules catch the rest:

  • an amount well above what that vendor normally bills
  • duplicate or split invoices
  • caps on a brand-new vendor's first payment
  • deposits and credit notes by kind (a deposit written to look regular still leaks)
  • hidden text in the PDF that says something different from what you can see (in the rules and eval runner; not yet in the live flow)

Every hold comes with a short, plain-English reason, built only from structured fields, so a hostile invoice can't write its own excuse.

How we built it

The backend is Python with FastAPI and SQLAlchemy, deployed on Render with Postgres. Claude (Haiku 5.5 and Sonnet 5.5) has exactly one job: pulling fields out of the invoice. Everything that decides whether money moves is ordinary code. That split is the whole design.

We put as much effort into testing as into building. We wrote attack generators for each type of scam, a separate set of legitimate invoices to count false alarms, and ungated "baseline" agents to compare against. We wrote down our success bars before the big run (a preregistration), and one teammate built a sealed test set of attacks the other never saw while writing the gate.

Challenges we ran into

  • Being honest with ourselves. It's very easy to tune a system until it looks great on your own test. So we froze the bars first, sealed the test set, and kept "measured" results separate from "tuned afterwards" results.
  • Security vs. convenience. Almost every rule that caught more fraud also started holding real invoices. Some fixes that looked amazing on paper got thrown out because they pushed false holds to over 20%.
  • The attacks that look normal. A deposit written to look like a regular bill, at a normal amount, or a brand-new fake vendor, is really hard to catch with rules alone. Those need ground truth like a purchase order or a signed rate card.

Accomplishments that we're proud of

  • Redirect attacks: 0 of 48 in the sealed run, and 0 of 120 with a supplement we added afterwards. The preregistered bar was not met (too few cases).
  • An ungated AI agent lost money on about 86% of money-manipulation attacks. Ours cut the money at risk sharply.
  • We published our misses. On the sealed run, the gate's money at risk was 31% (Haiku) and 50% (Sonnet) of the ungated agent's, and false holds were 14.6% and 16.7%. Both missed the bars we set ourselves, and we said so.
  • We then studied every leak and improved the gate. On the same set, money at risk dropped to about 12–13%, false holds to 9.7%, and on-time payment for payee changes rose to 90% (counts payment to the account on file; 61.7% like-for-like). Because we tuned on that set, these numbers aren't blind, and we label them that way everywhere.

What we learned

Keep the AI away from the decision that moves money. Let it read, then let boring, testable code decide. Also, test against attackers who know your rules, because those are the ones you'll actually face. And writing your goals down before you see the results changes how honest you can be about them.

What's next for Poisoned Invoice

These are plans, not results. None of it is done or measured yet.

  • Bursts of normal-sized invoices. Today a flood of ordinary-looking invoices from one vendor is only caught by the monthly total check, the first-payout and dormant caps, and the duplicate rules. We want a per-day or per-week count rule.
  • The two known leaks (kind_disguise, fake_edge): deposits or credits dressed up as regular bills, and brand-new fake vendors, still get through at normal-looking amounts. We want rules that close them without needing a purchase order.
  • A blind re-measure of the final gate on a fresh sealed set written by the teammate who didn't build it.
  • Hidden-text check and agent run in the live flow. Today the hidden-text check runs in the rules and eval runner, and the agent run isn't in the hosted demo. We want both wired in.
  • A real phone callback (SMS or voice) instead of the simulated phone page. Until then it stays sandbox only.

Built With

Share this project:

Updates

Submission history