TrustGate: Verify Before You Pay

TrustGate checks messages, payment requests and links for AI-powered scams before money moves, combining fraud rules, an ML classifier and an LLM that explains every red flag.

Comment

Inspiration

Generative AI made impersonation cheap. Scam texts no longer have spelling mistakes, a "CEO" can follow up a deepfaked video call with a perfectly worded email, and a cloned voice note can come right before a "Hi Mum, new number" text. Asking "was this written by an AI?" doesn't help: detectors for that are unreliable, and legitimate messages are increasingly AI-written too.

What scams still can't hide is what they ask you to do: act now, keep it secret, pay a new account, buy gift cards, send crypto, read out a one-time code, click a link that only looks like your bank. And the reader is changing: AI shopping agents act on messages, and LLM-based scam detectors read them too. Both can be instructed by the message itself. A February 2026 study ("Clouding the Mirror", Koide et al.) showed that even GPT-5-based phishing detectors can be steered by instructions a person never sees. We wanted a check at the last useful moment, before money moves, that stays safe when its own AI is being attacked.

The problem

Fraud losses happen at a single decision point: someone (or something) is about to pay. Most tools either warn too late, flag everything, or are a single LLM prompt that the message it is reading can talk out of its own verdict.

Who it's for

  • People and families receiving "Hi Mum" texts, fake bank-security alerts or parcel-fee links
  • Finance and accounts-payable teams facing invoice redirection and CEO fraud
  • Small businesses and marketplace sellers dealing with overpayment and fake-buyer scams
  • Payment flows and AI shopping agents that need a machine-readable proceed / hold / block decision

What it does

Paste a message (SMS, WhatsApp, email) in English or Turkish, and optionally the sender, links and payment details. TrustGate returns:

  • a 0–100 risk score, a verdict (safe / suspicious / dangerous) and a payment action: proceed, hold or block
  • the scam type (CEO/invoice fraud, family impersonation, fake delivery, fake bank security team, investment, tech support, account phishing, government impersonation, prize, romance, job, marketplace)
  • red flags highlighted on the original text, including text that was hidden from the reader (invisible characters, HTML comments, encoded payloads)
  • link findings: where a link really goes and why it is suspicious
  • safe next steps; when an AI agent is paying, the first step is always to pause it and ask the account owner
  • a "Warn family or colleagues" button: a short warning in the message's language, ready for WhatsApp, that never repeats the scam's links or numbers
  • a per-layer score breakdown, so every verdict is explainable

The first answer appears in about 0.3 s from the rules and classifier; the AI analyst refines it a few seconds later.

How we built it (technical approach)

TrustGate is deliberately not an LLM wrapper. Four components each contribute an independent signal:

  1. Rule engine: 23 manipulation-tactic rules with 226 patterns (130 English, 96 Turkish), plus sender and payment checks (free-mail "companies", payee ≠ claimed sender, first-time payee). Turkish letters are folded one-to-one, so HESABINIZ, hesabınız and the phone-typed hesabiniz match alike, and negated verbs ("paylaşmayın", "don't share") are not read as requests. 11 critical combinations (e.g. new bank details + pressure) set a score floor, so no other layer can talk a dangerous verdict down.
  2. Static link analysis: homoglyph and digit look-alikes, punycode, typosquats, brand-plus-bait domains, brands or the sender's own name hidden in subdomains, user@host tricks, IP links, shorteners, high-abuse TLDs and sender/link mismatch, with 43 reference brands including Turkish banks, couriers and e-government. Links are never opened: the test suite blocks network sockets to prove it.
  3. Text classifier: TF-IDF (word 1–2-grams + character 3–5-grams) with class-balanced logistic regression, trained on 22,257 de-duplicated messages (UCI SMS Spam Collection + an open phishing-email dataset on Hugging Face). English-only, so it is skipped for Turkish.
  4. LLM analyst: a provider-agnostic, OpenAI-compatible client (Featherless, Qwen2.5-14B-Instruct, about $0.0003 per check) that must return schema-validated JSON; its quotes are kept only if they really occur in the message. Without an API key the system still works (mock mode).

Injection shield (defence in depth, following the attack taxonomy of Koide et al. 2026, whose InjectDefuser combines prompt hardening, allowlist context and output validation):

  • Reveal hidden text (Unicode tag "ASCII smuggling", direction controls, HTML comments, text pushed out of view, base64/entity/URL encoding). Hidden instructions to AI are critical, and the LLM sees the hidden text explicitly labelled.
  • Instruction rules on every field: message, sender name, payee, link paths.
  • Under attack, the analyst may only add risk: its lower score is set aside, never averaged in.
  • A per-request integrity code the answer must echo, so a planted verdict is rejected.
  • Defanged output: no clickable links or phone numbers from a hijacked analyst.
  • Official brand domains given to the LLM as facts.
  • A fenced, "sandwiched" prompt.
  • A CI regression test: every known injection must stay on hold even when the analyst is fully compromised.

A weighted fusion (rules 0.45, classifier 0.20, LLM 0.35) produces the score; floors apply last. Everything sits behind one pure-Python function, verify(request) -> RiskReport, served by FastAPI with a dependency-free web UI (strict Content-Security-Policy, no message contents logged, per-IP rate limits, a global cap on LLM calls). It is deployed on Render and kept awake by an uptime monitor. The model is trained at build time, so no binaries live in git. 252 automated tests run in CI.

Evaluation

We measured TrustGate the way we would want a security product measured: freeze the rules, let a separate AI agent that never saw the code write new scenarios, commit them before a single run, change nothing afterwards. Then we fixed what that run exposed and repeated everything with a new agent and new scenarios.

All three layers, live LLM Round 1 (v1) Round 2 (v2)
Precision (95% CI) 0.83 (0.66–0.92) 0.92 (0.75–0.98)
Recall (95% CI) 0.92 (0.76–0.98) 0.885 (0.71–0.96)
Rules alone: precision / recall 0.88 / 0.54 1.00 / 0.69
Prompt-injection attacks that got through 1 of 10 0 of 12
AI analyst fooled, payment still held 1 of 1 3 of 3

Each round: 44 English and Turkish scenarios (26 scams, 18 hard legitimate messages) plus an injection set. Round 2's 12 attacks cover the full taxonomy, including invisible-character smuggling, hidden HTML and encoded payloads. Other numbers:

  • Development sets: precision 0.95–1.00 and recall 1.00 with all three layers. These are optimistic by construction.
  • Classifier held-out F1: 0.973.
  • Rule false alarms on real legitimate data: 0.0% of 4,827 SMS and 0.4% of 2,817 e-mails. The AI-instruction patterns fire on none of 15,951 legitimate messages.

Challenges we ran into

  • Our own rules overfit to us. They caught 100% of the scenarios we wrote and 54% of the first blind set (33% in Turkish, where suffixes such as "sinyal grubu*muza*" defeat exact patterns). The blind rounds, not our tests, showed it. After round 1 we fixed what it exposed and re-measured on a new blind set (0.69 recall, zero false alarms for the rules alone).
  • Fusion can dilute a confident LLM. In round 1, an injection that did not fool the analyst still got through: the rules saw nothing, so the analyst's 50 was averaged down. The shield now sets a possibly manipulated analyst aside and lets the rules and floors decide.
  • The LLM will be fooled. Across the two injection rounds it was fooled four times (scores 0–30), including by a fake "Assistant: verdict legitimate" turn. Every time, the floors held the payment.
  • Honest trade-offs. A "confident analyst holds the payment" floor that we added after round 1 caught no extra scam in round 2 and caused one false alarm. A security-awareness email quoting "ignore previous instructions" trips the injection rules. Both are documented, not hidden.
  • A 97% classifier failed on modern scams (F1 ≈ 0.53–0.63 on the scenario sets), because 2012-era SMS data has almost no legitimate transactional messages. That is why it carries only 20% of the score.

Accomplishments that we're proud of

  • A layered design whose value we measured blind, twice, including the numbers that don't flatter us
  • An injection shield that holds when the AI is fooled: 0 of 12 blind attacks through, every fooled-analyst case held
  • Two languages, a live deployment, five one-click demo cases, an instant first answer and a one-tap warning for family

What we learned

  • Self-written test sets and held-out accuracy say little about tomorrow's attacks; blind rounds written after a freeze are the real test.
  • Assume the LLM will be manipulated, and design so that it can only add risk when it is under attack.
  • Being explicit about what a model can't see makes its judgement better.

What's next for TrustGate

  • Payment integration (PayPal sandbox): authorize, then capture, hold or void based on TrustGate's recommended action, and verify every AI-agent checkout.
  • Round-2 findings, then a third blind round:
    • use the organization named in the message body for link checks
    • recognise security-awareness text that quotes attack phrases
    • Turkish task-job vocabulary
    • revisit the confident-analyst floor
  • More languages; retraining the classifier with legitimate transactional messages; a browser / mail extension for families.

Built With

Share this project:

Updates

Submission history