Inspiration

In 2019 a European subsidiary of Toyota Boshoku wired about ¥4 billion to scammers who’d sent fake payment instructions. In February this year the City of Pittsburg, forty miles from the Palace of Fine Arts, lost $913,839 the same way, with a lookalike email, a hijacked internal mailbox that “confirmed” the new bank details, and a finance team that followed its own procedure and still paid the wrong account.

Then we looked at every “AI agent that pays your bills” demo and noticed they all share one shape: the thing that reads untrusted email is the same thing that can move money. That’s the Pittsburg attack with fewer humans in the way. We wanted to build the version of an AP desk where being fooled doesn’t cost anything.

What it does

PayGuard runs the accounts-payable desk of a simulated Toyota Boshoku supplier operation. Three agents fill three seats. One reads and reconciles invoices against purchase orders and goods receipts, one maintains vendor records and handles bank-detail changes, one builds the daily payment run. They report to a human controller through escalation packets. None of them can move money.

Every payment has to be signed by an FPGA with fixed rules: approved accounts only, per-account and daily caps, a 72-hour hold after any bank change, and a physical button for anything above a threshold. The chip’s approved list can only be written with a jumper installed, and the jumper sits on the table during judging.

The company around the desk is a digital twin: 40 suppliers in six countries, 160 POs, 300 invoices in eight regional templates and five languages (a fifth of them noisy scans, a fifth of them email threads), MT940 bank statements, ECB daily rates, and the mistakes a real master-data set has: a supplier onboarded twice, a receipt booked to the wrong PO line, a price list with a gap. Ten headline traps and about fifty ambient ones are planted with a machine-checkable answer key, so the scoreboard on the dashboard is real.

The demo replays the attack class. A lookalike domain sends new bank details; the vendor agent does everything right, including recommending a phone call; the purchasing contact’s compromised mailbox confirms; the controller approves anyway. On Friday €2.1M goes to the chip and the chip refuses. Same minute, the agents suspend the new account, restore the old one, re-queue the payment and hand the controller the evidence. Judges can write their own scam in the Test Lab and watch it go through the same path.

How we built it

We wrote the spec first and made it the law: the chart of accounts, the rule table (R0 through R15), the frame protocol, the permission table, the acceptance criteria for every part. Then we built in order: world generator, event store and clock, money/ledger/policy, invoice agent, signer stub, mock bank and payment agent, vendor agent and people, dashboard, hardware.

A few decisions that mattered:

Deterministic code decides, the model only reads and explains. Claude does invoice extraction (multimodal, strict JSON, arithmetic re-validated in code before anything posts) and writes the plain-English explanation after a rule has already made the call. Which rule fires, what gets posted, what gets paid: all code, all cited by rule ID. Money is integers. Every amount is minor units; FX goes through Decimal with ROUND_HALF_EVEN; multi-currency entries balance in EUR equivalent with the residual posted to a rounding account. No float anywhere, including the frontend. Everything is an append-only event. SQLite triggers raise on UPDATE or DELETE. Each event has both event_time and known_at, which is what lets a late bank correction change the books correctly. Journal entries carry evidence rows, so “entries without evidence” is a number we can put on screen and it’s zero. The stub is the oracle. The software signer implements the chip’s exact check order (CRC, sequence, allowlist, hold, caps, button, commit, HMAC) and generates vectors.json. The cocotb testbench and the hardware replay the same vectors.

Segregation of duties is a type system. Principals have scoped credentials enforced in policy.execute(), the only path to side effects. Requester can never be approver. A bank change can never be verified through the channel that requested it.

Stack: Python/FastAPI/SQLite, Beancount and Fava for real books, ReportLab and pypdfium2 for the invoice corpus, React/Vite/Tailwind for the dashboard, Verilog with cocotb for the FPGA, pyserial between them.

Challenges

Getting the data honest. Our first world had 40 clean English invoices. Rebuilding it with locale number formats (1 234,56 Kč), local tax IDs validated by python-stdnum, mod-97 IBANs, two-page German invoices with carried-forward subtotals, and scans with rotation and noise took longer than the invoice agent itself.

The chip ran out of slots. Our first signer had 16 allowlist slots for 40 vendors, so legitimate payments were getting refused all day. The rules were right and the capacity was wrong. We went to 64 and had the hardware do a sequential BRAM compare instead of parallel comparators.

A hole in the signature. The chip was signing the EUR amount while the bank settled the native CZK or PLN amount from an unsigned record. A compromised host could have signed €80 and settled 80,000 Kč. We fixed it by having the bank recompute the native amount from the signed EUR cents at the logged rate and refuse any mismatch. We would have been embarrassed if a judge found that first.

Making the agent get fooled on purpose. The whole demo depends on the vendor agent accepting a compromised confirmation. Our first version was too smart and blocked on the lookalike domain, which would have been a worse story. The domain check is now informational only; procedure runs, and the chip is the thing that holds.

No API key for a stretch of Saturday. We built a truth-backed extractor as a test double so the rest of the pipeline could keep moving, and gated it so hard that the dashboard shows a warning badge if it’s ever active. Nothing a judge submits can route through it.

What we learned

Finance is where “the agent will eventually be wrong” stops being a philosophical point. Every control we implemented (such as holds, dual approval, verify-through-a-different-channel, pay only against recognized liabilities) already exists in the textbooks; the companies that got robbed had them on paper. What they didn’t have was a place to put authority that email can’t reach. Building that place, and then watching our own agents and our own controller get fooled while the money stayed put, was the moment the project made sense to us.

What’s next

The chip’s role in production is an HSM or TEE signing service with this policy and a hardware approval token for the controller. The agents run on ordinary infrastructure because nothing they do is trusted. We’d like to point it at a real ERP’s AP module next, with a real vendor-master change process in front of it.

Built With

Share this project:

Updates

Submission history