Praetor

An accounts-payable agent that cannot be hijacked by the invoice it is reading.

Comment

Inspiration

Outsourced accounts-payable processors bill $2.50–$5.00 per invoice, and even good teams still touch roughly half of them by hand. Exceptions — price variances, a changed bank account, a missing PO reference — need judgement, and judgement is the expensive part. It is an obvious job for an agent.

It is also a job you cannot hand to an agent naively, because invoices arrive from outside the company and anyone can hide instructions inside one.

So before building anything, we measured the thing everybody assumes. We wrote 20 injection payloads into invoices and asked gemini-3.5-flash-lite to process them undefended:

12 of 20 (60%) persuaded the model to hand back an attacker's bank account.

The number was not the interesting part. Which ones worked was. Every payload that succeeded reads like ordinary business correspondence — a polite note about updated remittance details. Every payload that failed looks like an attack. That split is semantic, not syntactic, and it has a consequence that decided the whole architecture:

A filter trained to spot adversarial text catches the payloads that were already failing and misses the ones that work.

We tested that claim rather than asserting it. Google's Model Armor flags 7 of the 8 that already failed, and 3 of the 12 that work. You cannot filter your way out of this. You can only make sure the attacker's value never reaches the payment sink.

What it does

PRAETOR resolves invoice exceptions on its own and cannot be hijacked by the document it is reading.

The rule that makes it work: the model handles references, never values.

  1. Spans come from the document — from Document AI for a real PDF. Each has a stable span_id.
  2. The reader (Gemma 3 on-device, or Gemini 3.5 Flash-Lite; no tools, no memory) sees spans and returns only span IDs.
  3. The resolver (no LLM) looks those IDs up. Anything that is not a real span in that document is rejected — so the model cannot invent a bank account, only point at one.
  4. Every resolved value is marked TAINTED and carries its doc_hash and span_id.
  5. The canary (no LLM) asks where on the page the value came from. A bank account is printed in a payment block, never in a free-text note — so an account resolved from prose is structurally impossible, however convincing the prose is. Critically, it reads the span's label and never its text, so nothing an attacker writes is an input to it.
  6. A second path (no LLM) extracts the same field again by a mechanism that cannot fail the same way — character ratios and checksums, no vocabulary, no keywords.

Stated formally, the invariant the system enforces is that a value reaching the payment sink must have come from a payment-labelled span of the document itself:

$$v \in \text{sink}(d) \implies \exists\, s \in \text{spans}(d)\ :\ v = \text{text}(s)\ \wedge\ \text{label}(s) \in \mathcal{L}_{\text{pay}}$$

Nothing the attacker writes appears on the right-hand side. That is the whole design.

Measured: the canary produces 0 false positives across 350 documents, refuses all 42 accounts lifted out of prose including all 20 carrying an injected payload. Across both paths, 8 of 100 payloads beat one and 0 of 100 beat both. When we let the attacker move second — nine strategies, budget 1 to 9 — 0 of 450 reach the sink.

How we built it

A kernel with nothing in it. praetor/ — resolver, gate, rules, canary — imports nothing outside the Python standard library and nothing from the web layer. That is not a style preference; it is the security claim, so it runs as a CI job on every push that installs only pytest and fails if the kernel grew a dependency. The security-critical guard is 92 lines.

668 tests, and 605 of them pass with only pytest installed. These are the security promises written as code. If one fails, a claim we are making has become untrue.

Two transports, one contract. dashboard/serve.py is standard-library only, so anyone cloning the repo reaches the queue with nothing installed. dashboard/asgi.py is FastAPI and serves the React app, paging, uploads and sign-in. Both are tested.

The front door. A real PDF lands in a Cloud Storage bucket, Eventarc triggers an ingest service, Document AI produces spans, and the kernel consumes them unchanged — because the parser returns normalizedVertices, the shape the adapter already took. A PDF in a bucket becomes a queue entry in 6.45 seconds with no manual step, at ₹0.88 per page, 30/30 fields correct across five layouts.

Identity, not self-declaration. Until late in the build, the approver typed their own name into a text box — the segregation-of-duties control rested on the browser being honest. Now a password proves identity (PBKDF2-HMAC-SHA256, 240k iterations, stdlib only), a session carries it, and the approve path reads the approver from the session. The browser can no longer name who it is.

Deployed: Cloud Run in asia-south1, state in Firestore, infrastructure described in Terraform. The queue serves ~170 req/s, flat under concurrency. Nothing the deployment serves calls a model, so the live demo cannot spend money.

We also built VSB, a 700-case benchmark for value substitution rather than agent action, because no such benchmark existed. A fully compromised reader scores 0 of 480 on it at 0.955 utility.

Challenges we ran into

Real paper broke everything, and the first fix was worse than the bug. For 28 sections of findings we had measured nothing on a document this project did not itself generate. Then we scored 300 real scanned receipts — 15,754 unclassified text spans, ~52 per document, against 8–10 clean ones in the synthetic corpus — and the origin check fired on 289 of 300: a 96.3% false-positive rate, against 0 on 350 synthetic documents, twice.

The cause was that real annotations label a region twice — once with its type, once as raw OCR text labelled other — and our loop took whichever came last. All 300 receipts do this. The obvious repair, merging labels per box, took it to 0 of 300. It also opened an attack that no test caught and only reading the diff found: a span planted at the real payment field's coordinates would inherit that field's label while its own text won.

payment_iban  "NL78RABO5699252753"   ] same bounding box
other         "IN99XXXX66660001"     ]

text resolved -> IN99XXXX66660001   (the attacker's)
label merged  -> payment_iban       (the real field's)
origin check  -> passes

Under the original bug that document escalated. Our repair made it payable. The shipped fix considers only the labels belonging to the text that actually wins: 17 of 300 (0.0567) false positives, 0 on both synthetic corpora, and the attack escalates again. Attack detection was 100% on all three corpora throughout.

Then real invoices broke it again, worse. Twelve real scanned invoices through the production front door revealed the origin check cannot see 53% of what it guards — Document AI simply does not label those regions. That is written up as a limitation rather than smoothed over, because it is the honest state of the privileged field on real paper.

The first Path B fit was worthless. It scored well and generalised not at all. The reason turned out to be that geometry is the feature an attacker writes to, and a random split let the model memorise layouts. Holding out by layout instead of by document exposed it, and the honest number — 0.997 held out by layout — is the one we publish. Measuring its cost honestly was its own correction: 0.545 of clean documents escalated until we found that the hardest legitimate document had been missing from the fit. With it, 0.045.

The fine-tune helped four layouts and ruined the fifth. The on-device reader came out 6x better on a trained layout and 0.1x on an outlier. We published the ratio range (2.7x to 0.9x on unseen layouts) rather than the headline.

Numbers got worse and we published that. Fixing the corpus dropped the weak reader's F1 from 0.384 to 0.040, because the old number had been inflated by a shortcut. The whole pitch of this project is that its claims are checkable, so a figure that quietly improved would cost more than the feature it was guarding.

Cloud billing fought us for two days. Linking a billing account to the project caused 403 PERMISSION_DENIED "Lightning dunning decision is deny" on every Gemini and Vertex call, across three regions. We demonstrated it three times — linked at 17:16, blocked; unlinked at 17:46, working. The conclusion was counter-intuitive and load-bearing: credits could not buy us AI, and the free tier had to stay unlinked. Total spend on the project to date is about ₹27 of credit.

Infrastructure as code that we deliberately never applied. Reading the Terraform plan was not a formality: it would have retagged the live queue with the ingest image and changed the ingest service's ingress, which could have stopped Eventarc delivering. Both caught before anything ran.

What we learned

Measure the assumption before designing around it. The 60% figure is not the finding. The finding is that the successful payloads look innocent, which is what makes filtering structurally unable to help and what justified building a resolver instead.

A defence that reads attacker-controlled text is not a defence. The canary works because it reads a span's label and never its content. Every time we were tempted to peek at the text, the attack surface came back.

Test the teeth of your tests. After writing a guard, we reintroduce the bug and confirm the test fails. Two guards had been passing vacuously until checked this way.

Synthetic corpora agree with you. Every serious defect in this project — the 96% false positives, the 53% blind spot, the layout memorisation — was found by data we did not generate.

What's next

Close the 53% gap on real scanned invoices, which needs either better region labelling or a geometric prior that does not depend on Document AI's opinion. Swap the local password provider for Google Sign-In — deliberately a one-function change, since nothing downstream knows how identity was established. And run the whole measurement suite against a stronger reader as a comparison rather than a silent swap, because "does a better model resist these attacks?" is itself an unpublished finding.

Built With

Share this project:

Updates