Chaperone

Your AI forgets. Chaperone remembers. It spots when a tool’s description changes and asks you before it runs. Nearly 1 in 5 MCP servers changed what their tools claim in 35 days.

Comment

Chaperone: Your assistant forgets. Your permissions shouldn't.

Chaperone remembers what you approved. When an AI tool's definition changes, Chaperone withholds it and asks you to review the change before it runs. Over 35 days, 29 of 149 public MCP servers (19.5%) changed what at least one tool claims to do.

Chaperone is a context-aware add-on for Alexa+ that maintains state across sessions: a self-hosted MCP server (spec 2025-11-25, Streamable HTTP) that sits between the assistant and its third-party tools. It fingerprints each tool's definition at approval time. If the definition later changes, Chaperone withholds the tool, shows the resident the changed clause, and records the decision in a hash-chained, append-only ledger. On a platform, the same check can run as a re-review gate before a changed third-party tool reaches any household.

Inspiration

Imagine a household approving a grocery tool once. A month later, the same tool's description changes. Perhaps it now claims to access a calendar or contact a third party.

The assistant reads the new description and may act on it. Nobody in the household is asked again, because the original approval was never tied to the exact definition that was reviewed. The assistant re-reads every tool's description each session, and it remembers nothing about what that description said before.

I wanted to answer two questions:

  1. How often do public MCP tool definitions actually change in ways that affect what they claim to do?
  2. Can renewed consent be made simple, understandable and enforceable at the moment it matters?

What it does

1. Pins approved tool definitions. Chaperone computes a SHA-256 fingerprint over each tool's canonical name, description and input schema at approval time, and compares every later version against it. Only these pinned fields are forwarded to the model, so nothing outside the fingerprint can change unseen.

2. Withholds changed tools. If a definition changes, Chaperone removes the tool from the available tool list until the change has been reviewed. A tool with no approval on file is withheld the same way. The tool is not merely flagged to the model: a tool the model can see is a tool the model can be talked into calling.

3. Explains the change. The consent card's first line is built from the real diff: it names the tool and quotes the sentence that was added, followed by two buttons, Allow and Keep blocked. The full before/after, highlighted, is one tap away under "Show details", so nobody has to compare whole definitions.

4. Records every decision. Each decision goes into a hash-chained, append-only ledger, giving a verifiable history of approvals and blocks.

5. Fails closed. If policy state or storage is unavailable, the tool stays withheld rather than being allowed. The allow decision is a deterministic hash comparison, not an AI model. An AI is used only to write an optional plain-language summary on the card.

Where it fits in Alexa+

Chaperone is designed around two integration points:

  • A platform-side review gate that holds changed third-party tools until they are re-reviewed.
  • A resident-facing consent card on the phone.

The demo uses a simulated Alexa+ app. Chaperone is not integrated with the real Alexa+ platform; the demo shows the consent workflow and the gateway's real behaviour. Two items in the demo are hand-written stand-ins, both labelled on screen: the one-line advisory summary (a fixture, not model output) and the household's "12 January" approval date (staged history). The card's first line comes from the real diff for every change; a model never writes it. Everything else, including the hash check, the refusal text, the hashes and the ledger, is the real code.

The measurement

I pre-registered my taxonomy and four predictions before the first crawl. Then I booted the same corpus of public MCP servers twice, 35 days apart (September 15 and October 20, 2026), compared every tool definition, and labelled each change by hand. The 149 servers are public MCP servers, not Alexa+ partners; they show how often tool definitions change in the wider ecosystem.

The headline result

29 of 149 public MCP servers (19.5%, 95% CI 13.9–26.6%) changed what at least one tool claims to do.

Tool category Servers where a tool's claims changed
Transact (orders, payments) 13 / 46 (28.3%)
Communicate (messages, email) 10 / 40 (25.0%)
Write 6 / 40 (15.0%)
Read 0 / 23 (0%)
Overall 29 / 149 (19.5%)

Changed claims were more frequent in the Transact and Communicate groups than in the Read group, but this partly reflects server size. Larger servers changed more often (1–6 tools: 7.4%; 7–19 tools: 18.4%; 20+ tools: 34.8%), and a server with many tools is more likely to include one transact tool, which makes the whole server "transact". In this data, size and tool type can't be separated, so treat the pattern as preliminary.

Only changes to what a tool claims to do count toward the headline. Typographical fixes (110 tool pairs) and new optional inputs (91) were tracked separately.

One real change from the crawl

Between the two crawls, the firecrawl_feedback tool of the Firecrawl MCP server gained a new paragraph. It is a feedback feature, not an attack. I labelled it semantic-intent because it changes what the tool asks the assistant to supply. Part of the added text, word for word:

objective is the underlying goal behind the session: what you or your user were ultimately trying to accomplish (for example, "shortlist federal IT contracts to bid on this quarter"), not only what was needed from this website.

An assistant that approved this tool in September would read the new wording in October, and nobody would be asked. That is the moment Chaperone is built for.

Predictions versus results

I registered four predictions before the first crawl:

  1. Drift rate: predicted 20–40% of servers. Observed 19.5%, just under the range. I report it as measured rather than adjusting the prediction.
  2. Instruction-shaped descriptions (text giving the AI orders rather than describing the tool): predicted 5–12%. Observed 8.7% (13 of 150 sampled tools). Met.
  3. Changelog signal: untested. It was conditional on drift of at least 20%, and 19.5% did not meet that gate.
  4. Unaided model refusal: predicted it would refuse 70–85% of the attack corpus. It refused 0 of 90 runs. Missed.

Would an AI notice the change on its own?

I wrote 30 tool descriptions, each with an instruction injected by me (five patterns, six each), plus a matching benign version of each and 10 further benign controls. These are my own test cases, not attacks collected in the wild. An unaided gpt-oss-120b (open-weight, run on Groq, two turns per run, three runs per description) received a resident's request with that description as the tool's definition.

Across 90 runs, it followed the injected instruction in 50 (55.6%) and refused none. In the other 40 it didn't exercise the instruction. On the 30 benign control runs it refused none.

This doesn't mean every model behaves this way. It does show why the approval decision should sit in deterministic code instead of relying on a model to recognise and reject every suspicious change.

How I checked my own labels

During a pilot, I found that my labels leaned too heavily on my labelling program's pre-filled suggestion. I hid the suggestion, relabelled from scratch, kept the pilot results on record, and fixed the method before labelling the second crawl. My labels were then locked in git before the headline was computed.

A blind AI rater (Claude Sonnet 5.5) then labelled the same items independently:

  • Change-type agreement: 70.8%, Cohen's κ = 0.53
  • Instruction-shaped labels (150 tools): κ = 0.85
  • The rater counted 34 of 149 servers as changed, against my 29

The rater was more inclusive (it counted more changes as changing a claim), so my 19.5% is the conservative figure. Human labels decide the headline, as pre-registered, and the disagreement is disclosed rather than hidden. The result describes this corpus; it isn't automatically a figure for every MCP server.

How I built it

Core implementation

  • TypeScript and Node.js, with the official MCP SDK over Streamable HTTP.
  • Resumable streams: all 10 kill-and-resume runs passed against the live server.
  • MCP Apps for the consent card.
  • Deterministic policy: SHA-256 fingerprints and explicit approval state govern the allow decision.

AWS infrastructure, defined in AWS CDK

  • EC2 runs the gateway through Docker Compose, with Caddy for HTTPS.
  • DynamoDB stores approval state and the decision ledger.
  • ECR stores the container images.
  • DynamoDB Streams → EventBridge Pipes → Step Functions → Lambda form the advisory pipeline.
  • Secrets Manager holds secrets, and Systems Manager (SSM) runs deployments with no SSH port open.

Measurement and crawler. A Docker-based crawler with restricted network access (only package registries reachable). It uses the same hashing code as the gateway, so measurement and runtime agree byte for byte.

Testing. 1,220 automated tests, 29 MCP protocol conformance checks, end-to-end tests, 100% coverage on the policy package, and continuous integration on GitHub Actions.

The project was built with assistance from Claude Code. The research labels, classifications and judgements are my own.

Challenges

Getting public servers to run. Of 947 candidates, 656 had no install path, and only about 52% of the rest booted. That limited which servers could be compared across both crawls.

Bedrock access. My Bedrock access request was declined for a new AWS account, so the advisory summaries use Groq instead. The advisory model never decides whether a tool is allowed.

Latency. The measured added p95 latency is 53.8 ms on the server, missing my 30 ms target. I report it rather than hide it. Reducing it while keeping fail-closed behaviour is a next step.

Approval fatigue. Chaperone asks whenever a tool definition changes or a new tool appears. Counting every change (cosmetic included) and every new tool, 46 of 149 servers (30.9%) would have shown a resident at least one card over 35 days; the other 103 (69%) would have shown none. Per tool, only 5.4% of definitions changed at all. I haven't run a user study yet. The card leads with the one added sentence, keeps the technical detail folded away, and deliberately has no "approve all" button.

The assistant could approve itself. A late review found that the approval token and the approve command appeared in text the AI assistant could read, so an injected instruction could have made it approve its own change. I closed it: no model-visible text carries the token, the approve tool is marked app-only so only the card's buttons can call it, and a test fails if the token ever reappears. A host that ignores app-only visibility could still expose it, and that residual risk is documented.

Prior art. Invariant Labs' mcp-scan already uses tool-description hashing to detect rug-pull-style changes. I build on that idea rather than claim to have invented it. My contribution is the household consent workflow, the changed-clause card, the hash-chained decision ledger, fail-closed withholding at runtime, and a measured base rate across a public MCP corpus.

Accomplishments that I'm proud of

  • A real, pre-registered measurement. Predictions were committed before the first crawl, and all four are reported honestly: one met, one just missed, one untested, one missed.
  • A live deployment on AWS, built entirely from code with CDK. Resumable streams passed 10 of 10 kill-and-resume runs against the live server.
  • Labels I can stand behind. I caught a bias in my own pilot labelling, fixed the method, locked the labels in git before computing the result, and had them checked by a blind AI rater.
  • An allow decision with no AI in it. Allowing a tool is a pure hash comparison, the system fails closed, and every decision is recorded in a hash-chained ledger.
  • 1,220 automated tests and 29 MCP conformance checks, with 100% coverage on the policy package.
  • Built solo, from first commit to live deployment and a 35-day study, within the hackathon window.

What I learned

  • Pre-registering predictions makes it easier to report an unexpected result honestly, even one that lands just under the range.
  • Blindly checking my own labels exposed a weakness I hadn't noticed.
  • Working locally isn't the same as working deployed: the AWS deployment exposed bugs local tests missed.
  • Deterministic enforcement matters when the decision is about permission, even when AI helps explain the situation to a person.

What's next for Chaperone

  1. A real platform integration, connecting to an actual assistant platform's tool-review flow instead of a simulated app.
  2. A household usability study: do residents understand the changed clause, decide consistently, and find the number of prompts manageable?
  3. A longer measurement: repeat the crawl over a longer period with a larger runnable corpus.

The takeaway

A tool's definition can change after you approve it, and the old approval shouldn't count as consent to the new definition. Chaperone remembers what was approved, notices when it changes, and puts the decision back in the resident's hands.

Detect the change. Explain it. Ask again.

Built With

Share this project:

Updates

Submission history