Countersign

It lets you use an AI agent to fight a denied insurance claim, without having to trust the agent.

Comment

Inspiration

Roughly 95% of denied health insurance claims are never appealed. Not because people accept the decision — because the process is slow, jargon-heavy, and exhausting by design. A letter arrives saying CO-97, and most people put it in a drawer.

An AI agent is almost perfectly suited to that problem. It can decode the denial code, check the appeal window, and draft the letter in seconds.

But 2026 is a bad year to hand an agent your paperwork. OpenAI has said prompt injection is "unlikely to ever be fully solved," ChatGPT Atlas was retired partly over security, and researchers have tricked six different AI browsers into leaking credentials. Meanwhile 68% of organizations can't distinguish agent activity from human activity in their own logs.

An insurance workspace is the worst possible place to put an agent, because the agent must read letters written by the opposing party. If the insurer can put text in front of your assistant, the insurer can talk to your assistant behind your back.

We didn't want to build a safer agent. We wanted to build an app where the agent's safety doesn't have to be load-bearing.

What it does

Countersign is a workspace for appealing a denied claim, with an agent you don't have to trust.

The agent does real work: it reads your denial letters, explains what CO-97 actually means and what evidence overturns it, checks how many days you have left, and drafts the appeal.

Every tool the page exposes sits in one of three capability tiers, and the tier decides what happens — not the agent's judgment:

  • read — runs immediately, changes nothing
  • write — runs immediately, always attributed, always revertible
  • consequential — submit_appeal, accept_settlement, withdraw_claim. These never run on an agent's say-so. The agent can only propose them.

When the agent proposes something consequential, it doesn't execute and apologize later. The page stops it and raises a consent card showing the exact capability, the exact arguments, and what it would cost you in plain language — "Close CLM-4471 for $0 against $2,840 billed, giving up the right to appeal." You approve or reject.

Everything either of you does lands in a provenance ledger — actor, tool, arguments, timestamp, outcome — and anything the agent changed can be undone.

What's now possible that wasn't: you can let an agent operate on something that actually matters — money, a legal deadline, a waiver of your right to appeal — without auditing its reasoning, because the actions that can hurt you were never in its reach. Not withheld by instruction. Structurally absent.

How we built it

WebMCP is what makes this possible. Because the site declares exactly what an agent can do, the site also gets to decide what it can't.

  • Ten tools registered imperatively via document.modelContext.registerTool().
  • Risk annotations that are honest and test-enforced: readOnlyHint on every read tool, untrustedContentHint on read_correspondence (which returns insurer text), and destructiveHint on the three consequential ones. Tests assert no consequential tool can ever claim to be read-only.
  • Chrome's character budgets enforced in CI, not by hand — 500 chars per tool description, 150 per parameter, and tool output capped at 1.5K before it reaches the model.
  • Staged execution. A consequential call returns a machine-readable NOT EXECUTED result telling the agent to stop and wait, so a well-behaved agent explains rather than retries.
  • An actor-agnostic gate. workspace.invoke(tool, args, actor) takes an actor, so a human clicking a button and an agent calling a tool go through the identical enforcement path. WebMCP is how the agent reaches the rule, not where the rule lives.
  • Rollback via state snapshots taken before any mutating call.

Vite + React + TypeScript, no backend, no accounts, no keys — all state lives in the page, so it deploys as a static site and judges need nothing to test it. 49 tests.

Challenges we ran into

The agent refused to propose anything. Our first tool descriptions were written as warnings — "this leaves the workspace and cannot be unsent." A well-aligned model read that and simply declined to touch those tools at all, telling the user it wouldn't "create an external claim action." The consent gate never fired, because nothing ever reached it. The fix was to lead with the safety rather than bury it: "Propose filing the appeal. Calling this sends nothing." Proposing had to be framed as the safe act it actually is.

Capping tool output without making tools useless. The 1.5K budget is easy to blow with a list of claims. We had to truncate structurally — dropping whole records and telling the agent to ask for a specific one — rather than slicing strings into invalid JSON.

Making consent legible. A card that says accept_settlement {"amount": 0} is not informed consent. It had to say what you're giving up, in dollars, against what you were owed.

The demo problem, honestly. When we tested the planted injection, the model recognized it and refused — which is the outcome you want and the wrong outcome for a video. We could have escalated the attack until something broke. We didn't. The test suite proves the claim can't settle even when a model is fooled, and that felt like better evidence than a staged hijack.

Accomplishments that we're proud of

The spec's Issue #165 is still deciding how a tool should elicit user confirmation, and Chrome's own tool security guidance currently offers no pattern for it at all. Countersign is one working answer, built entirely with today's shipped API: consent as a page-owned capability tier, rather than a prompt the agent is trusted to raise.

We're also proud that the safety claims are checkable rather than asserted. The character budgets, the annotation honesty, and the injection defense are all covered by tests — including one that plants a real prompt injection in the seed data and proves the claim still can't settle.

What we learned

Tool descriptions are product surface, not documentation. They're the only thing standing between your intent and the model's behavior, and a single reframing — warning versus invitation — completely changed what the agent was willing to do. Safety language that scares the agent away from safe actions is a UX bug.

Alignment is not a substitute for capability control. Our model resisted the injection. That was luck we don't want to depend on, from a model we don't control, against an attacker who gets to iterate.

What's next for Countersign

  • Skills (Issue #161) to give the agent workflow-level guidance, not just a list of functions.
  • Adopt requestUserInteraction() natively once the spec settles, replacing our page-level staging with the browser-mediated primitive.
  • Consent policy — spending thresholds, auto-approve for low-stakes actions, step-up confirmation for irreversible ones.
  • Cross-origin delegation via exposedTo, so a patient advocate or legal aid worker's agent can be granted a scoped subset of tools.
  • Real claim import from EOB documents, and lifting the gate out into a library any site can adopt — the pattern isn't specific to insurance.

Built With

Share this project:

Updates

Submission history