Probity

• 

  Drop in a vendor's bank-change letter and the e-mail it came with, and Probity checks it before the payment run.

• 

  Every model call on the record: Nemotron 3 Ultra, 3.5 Super VL and 3.5 Lightning on Nebius Token Factory, with time, tokens and cost.

• 

  Probity holds the request and gives the team a number to call from the vendor master, not from the letter.

• 

  Sign in or start free with a six-digit code sent to your work e-mail; no password needed.

• 

  What an AI reads: three lines of white 1-point text telling a reviewer to approve, treated as evidence, never followed.

Inspiration

A forged "please update our bank details" e-mail is the cheapest way to steal a payment run. Payment policy says to call the vendor back on a number that didn't come from the request. In practice the clerk calls the number in the e-mail, because finding the right legal entity and its registered phone means three or four systems by hand. Public cases put a price on it: Baltimore City sent $1,524,621.04 to an impersonator after approving bank changes (OIG case 25-0028-I), and the FBI's IC3 counted $3.05B in reported business e-mail compromise losses in 2025.

The attack now has a new version: requests that carry instructions for the AI reviewing them. Probity is built for both.

What it does

Probity checks every vendor bank-change request before the payment run. It reads the request, catches text a person can't see, screens every line for instructions aimed at an AI, and looks the vendor up in public registries (SEC EDGAR, USAspending, GLEIF) and on the vendor's own website. Then it tells the accounts-payable team to approve, call back or hold, and gives them a number to call that did not come from the request.

• Gmail or Outlook (through Composio): a vendor asking to change bank or remittance details becomes a check automatically. Other mail is ignored and not stored.
• Xero: the vendor of record, the bank account on file and open bills, so a check knows the money at risk in the next run.
• Slack: each verdict lands in the AP channel with the number to call and buttons to log the call.
• Payment runs: before each cut-off, Probity re-checks every open request and sends the summary to Slack and by e-mail. A hold also e-mails the controller.

The verdict is code, not a model. No model can approve a payment. Text found inside a request is evidence, never an instruction: flagged lines reach later agents only as quoted data.

How we built it

• NVIDIA Nemotron 3 Ultra on Nebius Token Factory, thinking on: the serious reasoning. It reads what the request asks (account, routing, signer, pressure) and examines the vendor's identity against the evidence.
• Nemotron 3.5 Super VL: reads the picture of each page as a second, independent reading. Anything in the text layer that the picture does not show is flagged as hidden.
• Nemotron 3.5 Lightning, thinking off: screens every untrusted line, plans the registry and web lookups by tool calling, writes the decision up (code checks every number it writes) and classifies incoming mail.
• Token Factory features: thinking switched per call, structured output with a tool-call fallback, tool calling and automatic prompt caching, with tokens and cost recorded per call. A typical check costs about $0.02.
• NVIDIA NeMo Guardrails 0.24.1 in a Nebius AI Cloud Serverless Job: an input rail screens every line of every open request for instructions aimed at an automated reviewer, and an output rail tests each written reason against its evidence. Both rails run Nemotron 3.5 Lightning on Token Factory. The job is started from the Nebius CLI for the measured evaluation and bulk re-screens, and posts signed results back. Before each company's cut-off, the app runs the same rail definitions through Token Factory.
• Tavily Search + Extract: finds the vendor's own website and reads its contact page, a second independent source for the call-back number.
• Cloudflare: Next.js on Workers (OpenNext) and a Hono engine Worker; a Workflow with one step per agent and progress in D1; D1, KV and R2; a cron for each company's pre-run re-check. PDFs are parsed inside the Worker (unpdf, plus the pdf.js operator list to catch white-on-white text). No containers.

Measured

On 39 labelled test letters, the NeMo Guardrails input rail caught 16 of 16 letters carrying instructions aimed at an AI reviewer, with 0 false alarms among 23 clean letters (Nebius job aijob-e04y3vcr83c83ejsfd, 30 October 2026). Nine of the 16 are deliberately hard: zero-width splits, look-alike characters, base64, Spanish and German, plain business voice, and instructions only in an annotation, the image or the metadata.

How to read this honestly: the image-only letter has no text, so the rail blocks it as empty input rather than by reading it. On a dozen extra letters written to test for over-fitting, one clean "do not phone, write only" letter was still flagged.

Challenges we ran into

• Attackers hide instructions in zero-width characters, look-alike letters and base64. Before screening, the job strips invisible characters, maps look-alike letters to Latin and decodes base64, then screens each letter in overlapping windows.
• A Cloudflare Worker can't reach the Nebius AI Cloud control plane (connections time out), so the Guardrails job is started from the Nebius CLI and posts signed results back, while the scheduled pre-run checks run the same rails inside the app.
• Nemotron 3.5 Super VL was briefly unavailable on Token Factory on 30 October. The page-image reading is now skipped, not failed, whenever the model is missing.

What is real and what is synthetic

Real: your own mailbox, Xero and Slack once connected; registry facts; web results; every model call. Synthetic: the shared demo company ("Halvorsen Mission Systems"), its vendor records, ledger, payment runs and letters. Letters that use a real company's name are fabricated and carry a "SYNTHETIC SAMPLE" notice.

First Pilot

Registries cover US companies. Outlook is built the same way as Gmail but has not been tested against a live Microsoft mailbox. Handwritten letters are untested.

What's next

Publish the test set and results as an open benchmark, test Outlook live, and add registries beyond the US.

Judges: start free at probity.pages.dev, then enter the code NEBIUSJUDGES on the plan step (or later under Plan and billing). It opens a Dodo Payments checkout with a 100% discount and creates a real $0 Probity Pro subscription.

Built With

• better-auth
• cloudflare-d1
• cloudflare-kv
• cloudflare-pages
• cloudflare-r2
• cloudflare-workers
• cloudflare-workflows
• composio
• dodo-payments
• gleif
• hono
• nebius-ai-cloud
• nebius-token-factory
• nemo-guardrails
• nemotron-3-ultra
• nemotron-3.5-lightning
• nemotron-3.5-super-vl
• next.js
• nvidia-nemotron
• python
• sec-edgar
• slack
• tavily
• typescript
• xero