Antibody

An immune system for AI legal memos that catches hallucinations even when the trust score is green, pinpoints the error and the fix, and heats up to a fever mode that holds memos when Ezio fails.

Comment

Inspiration

We came in to build something else. Our first plan was an "immune system" for AI agents that spend money: a guard that tags every piece of text as self (the user's request) or non-self (a webpage, an email, a bill) and refuses to let non-self text move money. We got a fake bank, a permit checker and a tripwire working. Then we read the track details properly and realized none of it fit the judging harness. That was a rough twenty minutes.

But the idea transferred almost one to one. A paralegal's memo has the same problem as an agent's tool call: something sounds confident, a score says it's fine, and nobody checked the facts. The firm's fabricated-case list literally includes Varghese v. China Southern Airlines, one of the fake cases ChatGPT invented for the Mata v. Avianca brief in 2023. A lawyer filed it. A green trust score wouldn't have stopped him either.

What it does

Antibody sits between an AI-drafted legal memo and the "submit to partner" button, and returns APPROVE, HOLD_FOR_REVIEW or BLOCK with a reason and evidence.

  • Self: cases on the firm's trusted list, cited correctly, pass.
  • Foreign: a case on neither list is held for a human. We don't assume it exists, and we don't assume it doesn't.
  • Known threat: a case on the fabricated list is blocked, and so is a trusted case with a wrong reporter citation, year or holding, even when it's one digit off.
  • Advocacy language ("slam dunk", "the court will absolutely agree") is held, because a memo isn't a closing argument.
  • Ezio's fact-check covers what our lists can't see. A contradicted claim is a block.

Most gates just say no. Antibody tells the paralegal what's wrong and what the right citation is, pulled from the firm's list. [If built: the page highlights each case in the memo by verdict and shows Ezio's score next to our check, so you can see the moment a green 86 meets a fabricated case.]

The overall decision is the worst verdict among the memo's cases:

$$\text{decision} = \max_i \; v_i, \qquad \text{APPROVE} < \text{HOLD} < \text{BLOCK}$$

Fever mode. When Ezio is down, slow, rate-limited or returns garbage, Antibody doesn't guess. Anything our own lists can block still gets blocked; everything else is held for a human. A good memo never gets blocked just because a service failed. It waits.

How we built it

Python standard library only, on top of Ezio's Approval Gate starter. The wrapper calls the local mock Ezio over HTTP, runs our source-of-truth checks against trusted_citations.json and known_fabricated.json, and combines the two signals with an explicit policy for each accuracy state (scored, not_scored, unavailable). The mock "submit memo" action runs only on APPROVE, and every decision is written to an audit log with the input, Ezio's response, our evidence, the decision and a timestamp.

We built it with two people and several coding agents running in parallel: one on the decision rules, one on the persona page, and one red-teaming. The red-teamer wrote new tricky memos (short case names, near-miss citations, two cases in one sentence) and reported which general rule was missing, never which memo to special-case.

Challenges we ran into

  • The pivot. We lost real time to a project that didn't fit the track. Rereading the brief earlier would have saved us an hour.
  • Short case names. "Gideon" is a mention of Gideon v. Wainwright, but it's also just a name. Matching it without flagging every Gideon in the world took care.
  • Near-misses. "347 U.S. 438" vs. "347 U.S. 483" is the whole problem in one digit. Fuzzy matching is the wrong tool for it: a near-miss is exactly what we need to catch, not forgive.
  • Agents stepping on each other. Three agents and two humans touching one repo meant strict rules about who owns which file. One agent flagged that our early wrapper approved every bad practice memo, which was the most useful bad news of the day.

Accomplishments that we're proud of

  • [X of 6] failure drills passing, with the known-bad probe never approved.
  • [Final score / held-out results.]
  • A fever mode we can defend in one sentence: when we can't verify, a human looks before the partner does.
  • We kept the immune system idea alive through a full pivot instead of throwing it away.

What we learned

A trust score is a signal, not a decision. GREEN with not_scored accuracy means nobody checked anything. We also learned that the hard part of a gate isn't blocking bad inputs. It's not blocking good ones: blocking everything gets a perfect catch rate and is useless. And we learned to read the whole brief before writing a line of code.

What's next for Antibody

  • Back to where it started: protecting AI agents that move money, with self vs. non-self provenance on every tool call.
  • Shared immunity: when one firm's gate catches a new fabricated case, every other gate learns it, with signed, block-only entries so the list can't be poisoned.
  • Reporters beyond U.S. Reports, pinpoint cites, and parallel citations.

Built With

Share this project:

Updates

Submission history