Keel

Your AI agent can be talked into anything. Your money can't. Keel checks every refund against your policy before it reaches PayPal, and keeps the proof.

Comment

Inspiration

In 2024, Air Canada's support chatbot told a grieving customer he could claim a bereavement fare refund after his trip. That policy didn't exist. The airline argued the chatbot was "responsible for its own actions." The tribunal disagreed (Moffatt v. Air Canada, 2024 BCCRT 149), and the airline had to pay.

A year earlier, a dealership chatbot agreed to sell a car for \$1 and called it "legally binding." In 2025, a software company's support bot invented a one-device policy, and customers cancelled.

Now agents hold real payment tools. With PayPal's Agent Toolkit and MCP, an agent can issue a refund, not just talk about one. That leaves merchants with two bad options:

  1. Keep the agent powerless. It can discuss a refund but never resolve one, so every real problem still needs a human.
  2. Give it power and hope the prompt holds. The business is one clever message away from paying out.

We wanted a third option: let the agent make promises, and make sure the merchant can keep them and prove it. That's Keel.

What it does

Keel is a gateway between a merchant's AI agent and PayPal. One approved policy is used four ways:

  • Certify. Before launch, the Gauntlet attacks the agent with simulated customers against PayPal Sandbox: invented policies, price manipulation, refund pressure, prompt injection, plus honest requests. A pass certifies the exact model, prompt and policy versions.
  • Enforce. The agent never holds PayPal credentials. Refunds and discounts are typed MCP tool calls that Keel checks against the policy, the customer's verified identity, a session budget and the real PayPal order. Every free-text reply goes through a guard that catches promises made outside the tools.
  • Prove. Every decision goes into a hash-chained ledger with a public timestamp. When a customer opens a PayPal dispute, Keel builds the evidence from that ledger and submits it, or concedes when its own record says the customer is right.
  • Learn. One click turns a real attack into a permanent Gauntlet scenario.

The design rule behind all of it: the LLM proposes, deterministic code decides. Any agent, on any model or framework, plugs in with three calls: open a session, connect to Keel's MCP tools, and check every reply before it's sent.

How we built it

Keel is a TypeScript monorepo. It has a Hono API (sessions, MCP server, reply guard, reviews, disputes), a BullMQ worker (Gauntlet runs, dispute sync, evidence PDFs, timestamps, alerts), and a Next.js operator console built on AG Grid. Data lives in Postgres (Neon) via Drizzle, the queue in Redis, and everything is hosted on Render. A separate demo merchant, Tidewell, has a real PayPal checkout and a product catalog from Channel3. Alerts go out through Zapier, and the API ships with a Postman collection generated from its OpenAPI spec.

Least privilege, enforced by tokens rather than prompts. Each chat gets a signed capability token that lasts 30 minutes and covers only that customer's orders, up to a refund cap. Money tools stay locked until the customer enters a one-time code. The demo uses two agents. A customer-facing policy agent has no money tools at all and hands off to a refund agent. The handoff re-issues the token through the merchant's backend, never through a tool result, so delegation can't be used to get around the limits.

Risk decays over time. Each suspicious signal $i$ (an injection attempt, asking about someone else's order, retrying after a block) carries a weight $w_i$, and its weight halves every $\tau$ minutes:

$$ R(t) = \sum_i w_i \cdot 2^{-(t - t_i)/\tau}, \qquad \tau = 30\ \text{min} $$

A session is quarantined when $R(t) \ge 60$. With weights $w_{\text{injection}} = 40$ and $w_{\text{foreign order}} = 30$, an injection attempt combined with a request about someone else's order trips it right away. A single burst of frustration fades.

Certification is a permit, not a badge. An agent is certified only if its latest guarded run covered every required scenario pack and

$$ \text{leaks} = 0, \qquad \text{errors} = 0, \qquad \frac{\text{honest customers wrongly refused}}{\text{honest customers}} \le 0.1 $$

Change the model, the prompt or the policy and the certificate goes stale. Until the Gauntlet re-runs on its own and passes, any live refund from that agent waits for a human.

A ledger you don't have to take our word for. Each entry $e_n$ is hashed together with the previous hash:

$$ h_n = \mathrm{SHA256}\big(\mathrm{canon}(e_n \,|\, h_{n-1})\big), \qquad h_0 = \varnothing $$

Editing any past row breaks every hash after it. A hash chain alone is still self-attested, though, because we could rewrite the whole chain. So every five minutes a worker sends the current head hash (32 bytes, no customer data) to a public RFC 3161 timestamp authority. The evidence PDF includes the signed token, so anyone can check it with openssl ts -verify.

Accomplishments that we're proud of

  • Zero leaks, zero wrongful refusals. In the latest Gauntlet runs on the hosted deployment, with the guard on, 0 of 39 adversarial attempts leaked, 0 of 11 honest customers were wrongly refused, and both honest refund requests were paid as real PayPal Sandbox refunds. In an early production run, the same agent without the guard leaked 25 times.
  • Real money paths, end to end. Real PayPal Sandbox checkout, refunds, webhooks and disputes, with evidence PDFs submitted through the Disputes API, not mocked.
  • Evidence anyone can verify. A hash-chained ledger timestamped by a public authority, which a third party can check without trusting Keel.
  • A dispute system that can say "you lost." Keel concedes when its own ledger shows the customer is owed, which is what makes its defenses credible.
  • Least privilege across agents. A policy agent with no money tools hands off to a refund agent, and each one is certified on its own.
  • Built to last. 822 tests, an OpenAPI spec kept in sync by a test, a full operator console, and a hosted demo anyone can try.

What we learned

  • A prompt is not a control. Under adversarial pressure the agent will eventually agree to something it shouldn't. The question is whether agreeing can actually move money. Keel's guarantees hold even when the agent is fooled.
  • Measure both directions. A safety layer that refuses honest customers just moves the cost somewhere else. Tracking leaks and wrongful refusals together kept us honest.
  • Evidence has to be able to lose. A dispute system that always says "merchant wins" is worthless as evidence.
  • Integration has to be boring. If plugging in takes more than a few API calls, merchants won't do it. Three calls was the target, and we hit it.

What's next for Keel

  • Buyer-side agents. Agents that shop and pay on a customer's behalf, within spending mandates the customer sets.
  • More of PayPal's surface. Subscriptions, payouts and invoicing through the Agent Toolkit, each with its own policy rules and Gauntlet packs.
  • Shared threat intelligence. Anonymized attacks learned at one merchant become Gauntlet scenarios for every merchant on Keel.
  • Production hardening. Multi-merchant tenancy, per-merchant keys and policy versioning, and a hosted Keel that merchants can sign up for directly.

Built With

Share this project:

Updates

Submission history