Action Check

Deterministic validation for WebMCP agent actions. Stop trusting AI agent replies. Action Check independently verifies WebMCP tool execution.

Comment

Inspiration

I started this hackathon building the same kind of thing most entries are: a site that exposes tools to an agent. The first commit in my working history, was an OAuth recovery workbench with six registered tools. A day into it I threw that out, because of something in Chrome's WebMCP evals guidance. In the troubleshooting table, for the case where a tool's output is wrong, it asks: was the UI state correctly updated, and did the agent receive the right information about the side effect? I went looking for anything that answered that question by actually calling a tool and then checking the site, and the readiness scanners I found only check whether a site exposes tools at all.

When an agent calls issue_refund or add_to_cart, the reply is the only information it gets. If the reply says "done" when nothing happened, or when it happened twice, the agent cannot tell, and neither can the person who asked it. So I built the test.

What it does

Action Check calls a page's registered WebMCP tool the way an agent would, through document.modelContext.executeTool in real Chrome, and does not take the reply at face value. Before and after the call it runs an observe() function that reads the site's own state, whatever fits that site: a cart in localStorage, rendered rows, a read-only endpoint. The verdict comes from comparing what the tool claimed with what actually changed.

observe() is never given the tool's result. It gets the live page, the phase (before or after), the tool name, the input and the page origin, and nothing else. I did it that way because the obvious shortcut, when you are writing an observer late at night, is to read the reply and return whatever it says, and I did not want that to be possible even for me.

This is what WebMCP makes possible that scripted clicks did not. Before, checking a site's action from outside meant driving its UI, and what you ended up testing was the UI. Now the tool is an addressable thing with a name and a schema, so the check a developer runs from the command line is the same call the agent makes. The person's part gets smaller: approve the exact values once, on the page, and step back. The agent does the calls and the retries. The proof step reads a record neither of them wrote, so nobody has to take anybody's word for it.

How I built it

The live page registers three tools, stage_refund_comparison, issue_refund and prove_refund_comparison, and answers one question a payments team would ask before exposing a refund tool: can a retry refund twice? The state lives in a small Cloudflare Worker with a SQLite Durable Object per lane and run. /v1/invoke returns only a claim; /v1/observe returns the durable state, effect IDs and an evidence digest. Every trial starts with /v1/reset, which has to prove a zero-effect baseline, and the Worker caps each trial at two invocations per lane on its own, whatever the browser does.

To make the retry path run, the harness commits the first refund and drops the provider's acknowledgement on purpose. The agent gets a bounded error and retries with the same request ID. On the known-bad lane that is two refunds from two calls; on the protected lane the request ID is honoured and there is one. The person approves the exact payment, amount, currency and request ID first, with a button that is an ordinary page control, so an agent cannot press it.

The three tools register one after another under a single AbortController, with an abort check between each, and the page shows the count going up; it only says the tools are ready once all three are in. Taking the view down aborts that controller. The signal that execute receives is passed through to the fetch that posts to /v1/invoke, so cancelling a tool call cancels the request, and because a cancelled request is not proof of a cancelled refund the session treats it as uncertain and re-reads the ledger. Inputs are checked twice: each tool advertises a frozen JSON schema, and execute parses the same arguments again with a strict Zod schema before anything moves. A failed parse comes back as INPUT_MISMATCH naming the fields. Every error carries a code, a message and a nextAction, and the hints depend on state, so an agent that calls things in the wrong order is told what to do next. The tools carry readOnlyHint and untrustedContentHint.

The CLI came on 2 September. It launches the installed Chrome with --enable-features=WebMCP (Playwright's bundled Chromium does not have WebMCP), reads the page's tools through document.modelContext.getTools(), and invokes the target once or twice with identical input. The target's own description, schema and annotations from that getTools() response go into the run's proof JSON, so each run records the contract it was checking in the site's words. Chrome 152 wants the input as JSON text, which the native test had already shown on 1 September, so the CLI sends text first and falls back to an object. On 3 September I added the external-target mode, so the same check can be pointed at a page I have nothing to do with.

Challenges I ran into

My own checker being wrong. Against Google's public WebMCP demos I recorded eighteen runs and set six aside because I could not stand behind the verdict. Three were on the observer side: a count-based observe() cannot see a retry that mints a second confirmation ID under the same banner, or a cart count hard-coded to 1. Three were my claim heuristic: it read a playful success reply ("You can't see me") as a refusal, and it had no answer for a tool that aborts its own registration mid-call. Those six are documented in the repository and not published. A fourth heuristic gap turned up on one of the published runs and I did fix that the same afternoon: a reply shaped {"status":"error"} now counts as a refusal, with two unit tests.

A claim I had written in three places that turned out to be false. I had said the verdict "never comes from the tool's reply". In retry mode that is true, the verdict depends only on the observed delta. In once mode the verdict compares the reply's claim against the delta, which is how it tells a false success from an honest refusal. I corrected the page, the README and this text. What I can say accurately is that the observer never sees the reply.

A page can only see tools registered on its own document.modelContext, not another origin's, so there is no way to build "paste a URL and check it" into the web page. The check has to drive a real Chrome from outside, which is why the CLI is the real product and the page is the front door to it. I added a builder to the page that takes a target URL, tool name, JSON input and mode, and writes out a starter observe() module and the exact command, so the first run is not a blank editor.

Accomplishments that I'm proud of

The results on pages I do not own. On 3 September, in Chrome 152, I ran twelve recorded checks against seven of Google's own WebMCP demos, and five found a bug. The ticket-booking demo's select_showtime replied "You can now proceed to checkout" while the checkout section stayed hidden, because the tool re-renders the page after opening it. The sports storefront's add_search_result_to_cart, retried once with identical input, put two of the same ball in the cart. Luxe Leather's add_to_cart doubled the quantity and the subtotal on retry while the header badge still said 1. Smart Home's rearrangeDOMComponents replied "Dashboard successfully updated with requested components" and rendered nothing. zaMaker's add_topping duplicated the topping on retry.

The other seven runs passed and I published those too: three honest refusals, three that were idempotent, one positive control. The proof JSON for every run is in docs/evidence/.

The verdict names ended up more useful than I expected when I wrote them. Retry mode gives DUPLICATE_EFFECT, NO_EFFECT or IDEMPOTENT. Once mode gives FALSE_SUCCESS, SILENT_EFFECT, EFFECT_CONFIRMED or HONEST_REFUSAL. "Your tool claimed success and nothing appeared" is a bug report someone can act on.

What I learned

An agent cannot verify its own work by asking the tool, and no amount of prompting fixes that; the check has to sit outside. A verdict is only as good as what the observer reads, and I should say so. Chrome's implementation has quirks the docs do not cover yet (the JSON-text input, tools that navigate mid-call and leave nothing to observe), and a harness has to report those as harness errors, not dress them up as verdicts. And I should check my claims against my own source before I repeat them, because the one about the verdict survived three documents before I did.

What's next for Action Check

Declarative form tools that navigate on submit are reported as unsupported for now; that is the next thing. Then identity-aware observers, so a retry that mints a second confirmation under the same banner is caught. Then a GitHub Action, so a site can run its own tools through this on every deploy.

Trying it

The refund demo is a synthetic sandbox: the payment, the amount and every effect ID are fictional, the ledger is a Worker I run, and no money moves. Full steps for ChatGPT's in-app browser, Chrome 149+ with the WebMCP flag, and the simulated agent for browsers without it are in the testing instructions and the README. From a clone, after npm ci, with Chrome 149+ installed, this reproduces one of the Google findings and exits 1:

node bin/action-check.mjs run --url 'https://googlechromelabs.github.io/webmcp-tools/demos/smart-home/#/' \
  --tool rearrangeDOMComponents --input '{"componentIds":["nonexistent_widget"]}' \
  --observe examples/observe-smart-home-dashboard.mjs --mode once

Tests: 206 unit across 27 files, 11 for the Worker, 20 end-to-end and one native Chrome WebMCP test, the last two also run against the live URL. ChatGPT's in-app browser was verified on the 1 September build; the 3 September build has been verified in Chrome 152.

Built With

Share this project:

Updates

Submission history