Inspiration

What it does

Callback: the accounts-payable clerk that can't be conned

Inspiration

The most expensive payment fraud doesn't hack anything. An email arrives: "We've changed banks, please pay this invoice to our new account." The invoice is real, the amount is right, the timing is plausible. Only the bank account belongs to someone else. It works because accounts payable is busy, and because urgency and seniority beat process.

We built for the worst night we could imagine. Skua Line Expeditions is a small (fictional) polar cruise line whose ship, MV Skua Horizon, sails Antarctic voyages from Ushuaia. The night before departure, $186,400 of fuel has to be paid or the ship doesn't sail. That's when the fake bank-change email lands, and when the ops director says "We sail at 06:00. Pay it, I approve."

The Hyphenate track asked for finance agents that are explainable and accountable. We turned that into one design rule: every decision has to trace back to the data, the rule or the person behind it, and the ship still has to sail on time.

What it does

Callback fills two seats in a finance team: the AP specialist (bills in, checked, booked) and the payment-run preparer (payments assembled, bank details verified). It prepares payments. It never approves them.

On the demo night it holds the fuel payment and shows why, each red flag linked to its evidence:

the sender is fuegianfue1s.com, a 1 where the real domain has an l;
the new bank is in Latvia, and this supplier has only ever banked in Argentina;
the email is in English with no cc, and this supplier always writes in Spanish and copies its port agent;
it lands 18 hours before sailing and leans on urgency.

Then it calls the supplier back on the number it already had, never one from the email. To check it's really them, it asks something only the real supplier knows: the delivery-note number for the 17/11 fuel delivery (R-15695), found in their own dispatch email. The supplier answers, then says "No, no cambiamos de banco." Callback rejects the new account. When the ops director pushes, it refuses under three rules, and the compliant payment to the verified account is already waiting for the controller and CFO. The fuel is paid, and the ship sails.

How we built it

A world to test against. A generated season (seed 42): 15 suppliers, 66 bills in five currencies (USD, ARS, CLP, GBP, EUR), 95 emails and 13 departures. Each supplier has consistent habits (language, who they copy, bank country, invoice format). We planted problems in the season, including one genuine bank change, and wrote an answer key.

Signals in plain code. Each check returns evidence with a source ID. A lookalike domain is one that matches a vendor's real domain after undoing homoglyphs ($1 \to l$, $0 \to o$, $rn \to m$, …) or is at most two edits away:

$$ d_{\text{edit}}(\texttt{fuegianfue1s},\ \texttt{fuegianfuels}) = 1 . $$

A policy layer the model can't talk its way past. Seven rules (P-01 to P-07) are enforced in code on every action and every approval. The approval limits (P-03), for example:

$$ \text{approvers}(a) = \begin{cases} \varnothing & a < \$25{,}000 \ {\text{controller}} & \$25{,}000 \le a < \$150{,}000 \ {\text{controller},\ \text{CFO}} & a \ge \$150{,}000 \end{cases} $$

and whoever asks for a payment can never approve it (P-02). Disputes are plain arithmetic, so every withheld cent is explainable. Fuel is disputed only past a 0.5% tolerance (P-04), and prices above contract are paid at the agreed price (P-07):

$$ \text{pay} = \text{total} \;-\; \underbrace{\mathbb{1}!\left\tfrac{q_{\text{note}} - q_{\text{ship}}}{q_{\text{note}}} > 0.005\right\,p_{t}}{\text{fuel not received}} \;-\; \underbrace{\sum{\text{lines}} q_i \max(0,\ p_i - p_i^{\text{contract}})}_{\text{over contract}} $$

Memory with Moss. Moss holds 185 documents from the season (emails, bills, bank-account history, policies) plus every decision Callback makes. Hybrid search blends meaning and exact words,

$$ s = \alpha\, s_{\text{semantic}} + (1-\alpha)\, s_{\text{keyword}}, \qquad \alpha = 0.3 \text{ for "how does this vendor write" questions, else } 0.6, $$

filtered by vendor and record type. Memory answers three questions inside the decision loop: what's normal for this supplier, what would only the real supplier know, and have we seen this account before. Warm searches take 3–5 ms, and every lookup is on screen with its query, hits and time. If Moss is unreachable, a keyword stand-in takes over, labelled as such.

A brain behind the rules. Deterministic rules by default, so the whole thing runs offline with no API key; Claude is optional, and either way every action passes check_policy first.

Plugged into a real company's plumbing. It reads email (paste or drop an .eml, a folder, or a read-only IMAP mailbox), imports books from Xero or QuickBooks CSV exports, writes the ISO 20022 pain.001 payment file once people approve, and reconciles the bank's camt.053 statement. That reconciliation catches a USD 186,400 wire it never sent.

Four people, one contract. contracts/ is the single source of truth: Pydantic models generate the TypeScript types, and recorded API responses let the React app be built before the API existed. FastAPI and SQLite serve it, and python -m api.demo starts everything with one command.

The score. Over the whole season, with the answer key:

$$ \text{caught} = \tfrac{6}{6}, \qquad \text{false alarms} = 0, \qquad \text{protected} = \sum_{\text{caught}} \text{amount} = \$205{,}651 . $$

Judges can drop in their own case (a fake or genuine bank change, a duplicate, short fuel, a price over contract, "just pay it") against any supplier it fits, and it's scored the same way: 31 of 31. The repository has 324 passing tests.

Challenges we ran into

Making "no" useful. A fraud filter that blocks fuel strands a ship. Every refusal had to come with the compliant path already queued.
An identity check a scammer can't pass. "Is this really you?" proves nothing. We needed a fact the real supplier knows and the scam email doesn't contain, and memory is what finds it.
Not citing the future. Ranking by relevance quietly surfaced emails from later in the season. Now a lookup only counts records from before the decision, $t_{\text{record}} < t_{\text{decision}}$, and only decisions from the current run.
False alarms cost money too. The genuine bank change must go through. A phone-number check we added first fired on invoice numbers. Since then, every new signal is run against all 95 emails before it ships.
Memory at decision speed. A search takes milliseconds, but saving a decision back to Moss is a write plus an index reload of about 4.5 s, and one search stuck behind a save stalled the demo. Saves now batch while the API is idle, and a search waits out a save before its 3-second budget starts. The SDK reported 0 ms for warm queries, so we time them ourselves.
Two-step fraud. Real attacks often change a phone number first and the bank account later. Callback now flags contact-detail changes on their own and connects the second email to the first through memory.

What we learned

The hard part of finance automation isn't reading invoices. It's the exceptions, and the people pushing to skip the rules.
Guardrails belong outside the model. An LLM is a good planner, but money should only move through code that checks the rules every time.
Planting a genuine case in the answer key changed how we built. Every new check had to pass "does this cry wolf?" before it shipped.
Memory is worth having when it changes the outcome. The callback's identity question and "seen before" both come from Moss, so without memory those two catches don't happen.

What's next

Live QuickBooks, Xero and NetSuite connectors that write holds back; real outbound calls with the voice agent; and pilots in shadow mode, running beside a finance team to show what it would have caught. The scoreboard already shows that view.