COUNTERSIGN

Cryptographic trust for payment requests in an AI-compromised world.

Comment

Inspiration# COUNTERSIGN — Cryptographic Trust in an AI-Compromised World

Inspiration

We started building COUNTERSIGN around a simple concern: how can we trust a payment request when even the information explaining it could be manipulated?

Today, a message can look completely legitimate while containing fraudulent payment details. With AI making it easier to generate convincing explanations and messages, simply trusting what looks or sounds right is no longer enough.

That led us to an important question: what happens if the AI system responsible for interpreting a payment request is itself compromised? Could it convince the system to approve something that should never be trusted?

We wanted to explore a different approach. Instead of allowing AI to make the final decision, we designed COUNTERSIGN so that trust depends on cryptographic evidence and explicit security rules.

What It Does

COUNTERSIGN is a security-focused prototype that checks payment requests using cryptographic verification and policy enforcement.

It produces three possible outcomes:

  • VERIFIED: The relevant fields have been cryptographically verified under the configured trust and policy rules. This does not mean that every statement in the surrounding message is trustworthy.
  • CONTRADICTS_POLICY: The request conflicts with a configured security policy.
  • UNVERIFIABLE: The available evidence is not sufficient to establish the required trust.

The idea is to make the difference between verified information, policy violations, and uncertainty clear instead of treating every convincing claim as a trustworthy one.

How We Built It

One of our main design decisions was to treat AI as an untrusted sensor rather than an authority.

We built the prototype using Python for the verification logic, FastAPI for the backend API, and React, TypeScript, and Vite for the frontend. For cryptographic signatures, we used Ed25519 through PyNaCl.

The system brings together a few key components:

  • Cryptographic verification: Checks signed data and relevant payment fields against the configured trust information.
  • Policy enforcement: Applies explicit security rules, including restrictions involving sensitive credentials and suspicious payment instructions.
  • AI interpretation: Extracts claims from the request, but those claims remain untrusted until checked against independent evidence.
  • Evidence panel: Lets users inspect the information behind a verdict and understand what was established, what conflicts with policy, and what remains uncertain.
  • Adversarial testing: Tests how the system behaves when data is modified, AI claims contradict policy, outputs are malformed or unsupported, or the AI component behaves maliciously.

We used synthetic test scenarios and fictional entities so that we could explore these security problems without processing real banking transactions.

The Security Principle Behind COUNTERSIGN

The most important rule in our design is that a compromised AI must not be able to create trust simply by making a confident claim.

Even if the AI produces a convincing explanation or supplies attacker-controlled payment details, it cannot override cryptographic verification or independently authorize a VERIFIED result.

If the required evidence cannot establish trust, the system must not pretend that it can.

This separation between AI interpretation and security-critical decisions is what COUNTERSIGN is built around.

Challenges We Faced

One of the biggest challenges was making sure that untrusted AI output could never cross into the part of the system responsible for security decisions. It is relatively easy to demonstrate how a system behaves when everything works as expected. The harder part is checking what happens when inputs are manipulated or the AI component behaves in ways we did not intend.

We focused on keeping cryptographic evidence, policy decisions, AI-generated claims, and final verdicts separate.

During testing, we also encountered a frontend issue where structured evidence data caused the React interface to crash. We traced the problem to how the evidence values were rendered and fixed the formatting without changing the cryptographic verification logic. Afterward, we rebuilt the frontend and reran the automated tests.

These problems reminded us that security is not just about getting the main logic right. The way a system handles unexpected input matters just as much.

What We Learned

Working on COUNTERSIGN helped us understand why security-critical systems need clear boundaries between interpretation and verification.

We learned that:

  • A confident explanation is not proof that something is trustworthy.
  • AI-generated claims should never be confused with authenticated facts.
  • When evidence is insufficient, the system should make that uncertainty explicit.
  • Testing malicious and unexpected behavior is just as important as testing normal scenarios.
  • A security interface should help people understand the evidence without claiming more than the system has actually verified.

What's Next

COUNTERSIGN is a prototype, and there is still more work that could be done. We would like to expand the adversarial test scenarios, strengthen integration testing, and explore how easily people can understand the evidence and verdicts when using the system.

Our goal was not to build another AI tool that asks users to trust its answers. We wanted to explore how AI could be useful while keeping the responsibility for verification with cryptographic evidence and explicit security rules.

COUNTERSIGN — Trust the evidence. Never the confidence.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for COUNTERSIGN

Built With

Share this project:

Updates

Submission history