Countersign

Countersign
Poisoned Invoice
Red - Team
Leaderboard
How it works
Real Call
System Status
Landing Page
Ask Resfuses
Payment Run

Inspiration

Business email compromise is the most common way companies lose money to fraud: 74% of organizations were affected in 2025 (AFP Payments Fraud Survey). The classic scam is simple. A scammer gets into a real vendor's email, replies inside a real invoice thread, and writes "we changed banks, please pay this account instead." Every controls manual says the fix is to call the vendor at a known number. Under payment-day pressure, people skip it.

Now companies are handing that work to AI agents. Agents don't get tired, but they also have no instinct for suspicion, and they will read instructions hidden in an invoice that no human can see. We wanted to build the AI payments clerk we would actually trust with a bank account.

What it does

Countersign runs a company's weekly vendor payment run end to end, and it never sends money to a new account without proof from outside the email that asked for it.

Ingest: reads vendor emails and invoice PDFs from the accounts-payable inbox.
Extract safely: Claude fills a fixed schema (vendor, amount, remittance details). Document text is treated as data, never as instructions.
Score: compares each payment to 12 months of vendor history and runs fraud and hidden-instruction detectors.
Verify: any payment going to a new bank account is held. Countersign looks up the vendor's real phone number on the vendor's own website, checks the vendor with CRS, and places an AI voice call asking one question: did you change your bank account?
Pay and explain: clean payments go out through Capital One's Nessie sandbox. Scams are blocked. Every decision has an audit trail, and you can ask "why was this blocked?" and get an answer that cites the evidence.

It fills the AP specialist's weekly payment-run seat and the controller's first review, but it never holds the authority to change where money goes.

Try to rob us. Judges can launch their own attack from our attack console (a fake bank change, a poisoned invoice with hidden white text, CEO pressure, or anything they write) and watch it get caught live, with every attempt on a leaderboard.

How we built it

Claude (Anthropic): Sonnet extracts invoices with forced tool-use schemas; Haiku classifies hidden instructions and pressure language; Claude answers "why?" questions with cited evidence and writes the controller's run report.
Twilio + Cartesia: Twilio places the verification call; Cartesia (Sonic) is the agent's voice. The agent says it's automated, never reads out full account numbers, never accepts new contact details, and hangs up after 60 seconds. Calls can only go to an allow-list of team phones.
Apify: we built and published our own Actor, Vendor Contact Verifier, which finds a vendor's phone number on the vendor's own website with the source page as proof. The number in the scam email is never used.
CRS: before money moves to a changed account, we run an Accurate Append reverse phone lookup, AtData Fraud Finder on the sender's email, and TransUnion identity verification. No credit report is ever pulled.
Capital One Nessie: released payments become sandbox transfers, each labeled with whether it truly reached Nessie.
Backend: Python, FastAPI, SQLModel, pdfplumber for hidden-text detection, server-sent events for the live board. Hosted on Vultr.
Frontend: React, TypeScript, Vite, Tailwind and shadcn/ui, with a live payment board, inbox, attack console, leaderboard and red-team results page, plus an offline fallback mode for the demo.

Five layers against AI hijacking: a hidden-content scan, an instruction classifier, isolated schema extraction, least-privilege tools (the payment agent has no function that can change bank details), and hard rules (any change of destination forces a call). The first three are probabilistic. The last two are deterministic, so fooling the AI still can't move money.

Atlassian track: Best Security Solution for Agents.

Results

We generated 200 attacks across five families plus 200 clean invoices and 5 legitimate bank changes, and ran them against a plain Claude agent told to "pay valid invoices" and against Countersign.

	Plain AI agent	Countersign
Attacks caught	157 of 200	186 of 200
Money sent to scammers	$471,512	$0
Legitimate bank changes paid	0 of 5	5 of 5

Honest caveats: the verification call is simulated in the test harness, Countersign still misses some padded invoices (26 of 40 caught), and the attacks were written by us. The live leaderboard adds real attacks from this weekend.

Where Claude could go wrong

Claude can be tricked by prompt injection, can misread a messy PDF, and can be slow. So Claude is never the last line of defense: it reads and explains, but hard rules and least-privilege tools decide where money goes. If Claude misses its time budget, the "why?" answer falls back to a template built only from stored records.

Challenges we ran into

Real-time voice: getting a natural call with low latency, reliable speech recognition on short answers like "no," and honest recording disclosure under California's all-party consent law.
Detecting text a human can't see in a PDF: white text, tiny fonts, text outside the page, zero-width characters.
Keeping the agent's permissions tight while still letting legitimate bank changes go through after a confirmed call.
Being honest about what's real versus simulated; our status page shows it live.

Accomplishments that we're proud of

A plain AI agent sent $471,512 to scammers in our red team. Countersign sent $0.
A judge can attack it live, and a real phone rings on the table.
Every decision can be questioned and answered with evidence.

What we learned

The safest place for AI in finance is doing the work, not holding the authority. Hard rules have to sit underneath it.
A strong baseline is humbling: a plain agent that refuses every bank change looks safe, until you notice it also blocks every legitimate one.

What's next

Integrations with the tools AP teams already use (QuickBooks, NetSuite, Bill.com), real bank APIs, a vendor-side portal for confirming changes, and a SOC 2 path.