AgentPerf

What it does

AgentPerf answers the question every frontend team is about to be asked: if we build and maintain a second surface for AI agents, what do we actually get?

Three pieces answer it end to end.

agentperf, the benchmark CLI. Runs the same task, on the same page, with the same model, twice. Once by DOM driving, an accessibility-tree driver that reads, clicks, fills and selects, the way browser agents work today. Once through the page's WebMCP tools. Identical agent loop, identical prompt, identical verification. It reports wall-clock, tokens, round-trips and success rate, per run and aggregated.

@agentperf/react, the fast lane. Three primitives that turn an existing React app into a WebMCP surface without touching the human UI.

Two instrumented demo pages and a landing page, all live, all publishing their own tools. The landing page will hand your agent its own benchmark data with the caveats attached.

The measurement

gpt-5.6-luna, n=5 per lane, both lanes 100% successful on both pages:

page	accessibility tree	lane	wall-clock	tokens	round-trips
salon booking	1,010 chars	DOM driving	7.5s	10,079	5
salon booking		WebMCP tools	5.0s	4,268	4
72-product catalog	32,916 chars	DOM driving	24.3s	102,537	7
72-product catalog		WebMCP tools	6.0s	7,434	5

2.4x fewer tokens on the tiny page. 13.8x on the realistic one. Make the page 32x heavier and DOM driving pays 10x more, while the tools lane barely moves, because it never reads the page. It asks.

That is the finding, and it is not the one anyone is quoting: the gap is a property of the page, not of WebMCP. A single number for "what WebMCP saves" is meaningless without the page it was measured on. That is why this ships as a harness you point at your own site, not as a headline.

It replicates across model families: claude-sonnet-5 via OpenRouter, with no vendor-specific workarounds, pays 2.5x on the small page with identical round-trip counts.

Why this use case is a strong fit for WebMCP

Most WebMCP demos show that an agent can drive a site. The open question is whether a team should pay to build that, and nobody could answer it with data. The "10x faster, ~90% fewer tokens" figures in circulation trace to one methodology-free blog post and to token-only counts measured against screenshot baselines by the ecosystem itself. Google's own WebMCP Evals guide covers correctness (does the agent pick the right tool) and explicitly provides no metrics, no harness, and no comparison against DOM automation.

So the fit is direct: WebMCP is the thing being adopted, and the missing artifact is the evidence for adopting it. AgentPerf is built on WebMCP twice over. The demo pages publish tools through document.modelContext, and the harness implements the other half of the protocol, acting as a WebMCP host so it can measure any page that speaks it.

How it creates a better experience

For the person: the agent's work happens in their tab, in their UI. When it searches, the grid filters. When it orders, the confirmation appears where they can see it. A dock states plainly how many tools the page has exposed. Agent-readiness usually means a headless side door the user cannot observe; here it is a visible second doorway onto the same app.

For the developer: the fast lane is additive. Delete agent-surface.tsx from either demo and you have an ordinary React app. In a browser without WebMCP, everything no-ops.

For the agent: it stops guessing. It gets typed actions and a state snapshot pruned to a token budget, with explicit "+N more" markers wherever a list was elided, so a partial read is legible as a partial read instead of becoming a confident hallucination.

What people and agents can now do together that was hard before

• Buy correctly from a real catalog in four tool calls. On a 72-product store, "find the cheapest in-stock wireless keyboard rated 4.5 or higher and order it" is a filter, a comparison and a checkout. Through tools that is search_products, add_to_cart, place_order, for 7,434 tokens. By DOM it costs 102,537 tokens of reading and re-reading the same grid.
• Let an agent write to an app without letting it corrupt the app. Every action is zod-gated. A malformed email never reaches the handler; the agent gets back the exact violated field paths and a deterministic fix-and-retry instruction, and corrects itself in one turn instead of re-sending the same bad call.
• Ask a marketing page to justify its own claims. The landing page publishes get_benchmark_results. An agent reading it gets the numbers and the caveats (n, model, page sizes, known biases) instead of adjectives.
• Benchmark a WebMCP page you did not write. The harness supplies the host, so it works on any page that speaks the protocol, in stock Chromium, with no flag and no origin trial.

How I implemented WebMCP

Publishing side, @agentperf/react:

• useAgentState(key, value): every slice flows into a single get_page_state tool, serialized within a token budget (default 1,000) with deterministic pruning and explicit elision markers.
• useAgentAction(name, config): registers a tool whose input schema is generated from a zod schema via z.toJSONSchema, so the contract and the schema the agent sees cannot drift. readOnly and destructive map to WebMCP annotations. A reserved, currently inert price field anticipates x402 settlement without a future breaking change.
• AgentBoundary: scopes state keys and tool names the way the component tree scopes the UI.
• Registration hygiene: handlers, descriptions and values are read through refs at call time, so re-renders never tear down and re-register a tool. Each tool holds an AbortController and unregisters on unmount, which fires toolchange so an idle agent learns the surface moved.

Host side, the harness: WebMCP lives behind an origin trial and a Chrome flag, which makes reproducible measurement painful and CI impossible. So the harness ships a host shim and injects it with Playwright's addInitScript before page load. The page registers into that shim, and window.__agentperf.listTools() and .call() bridge those tools into the agent loop as function tools.

Verification is the benchmark. A run counts as a success only if the model calls task_complete and the rendered page matches the task's successPattern. For the catalog that pattern pins the product and the price, so ordering the wrong keyboard fails the run instead of passing on a bare order number. A task whose pattern already matches at page load is rejected outright as unverifiable.

Challenges I ran into

My own benchmark was unfair, and I published it before I found out. The launched numbers were 2.4x tokens and 2.6x wall-clock. Then I had the methodology adversarially reviewed, and the review was right: the DOM lane's click returned only "Clicked button X", forcing a separate read_page to see the result, while every real accessibility-tree driver returns the post-action snapshot with the action result. I had doubled that lane's round-trips by design, then billed it for re-sending the transcript each time. Fixed and re-ran: the token ratio survived unchanged at 2.36x; the wall-clock claim did not. 2.6x was wrong, it is 1.5x. The correction is on the landing page and at the top of the README.

Then it nearly happened again, in my favour. The first heavy-page run was spectacular: the DOM lane succeeded 1 time in 5, ordering a keyboard rated 4.2 when the task demanded 4.5 or higher, and announcing success every time. "DOM driving doesn't just cost more, it buys the wrong thing" is a far better headline than any cost ratio. It was also my bug: the catalog's rating filter is a select element and my DOM lane had no way to operate a dropdown. Adding select_option took it from 20% to 100%. Two rules came out of that: a lane is only a baseline if it can do what the real thing can do, and a result that flatters your conclusion gets audited harder, not published faster.

Rate limits were silently biasing the result. A 429 failed a run outright, which penalizes whichever lane spends more tokens, the lane under measurement. Retries now back off exponentially. Crashed runs also used to record zero tokens and get averaged in, dragging a failing lane's cost toward zero; central tendencies are now over successful runs only, with failures in the success rate.

gpt-5.6 returns a 400 on function tools unless reasoning_effort is "none". My first instinct, swallowing provider errors so a run doesn't abort, was exactly wrong for a benchmark: it turns a broken configuration into a plausible-looking failed run.

Accomplishments that I'm proud of

• Every number that moved, moved against me, and I shipped it anyway: on the landing page, in the README, and inside a WebMCP tool the page itself serves.
• benchmarks/METHODOLOGY.md: the full adversarial review of my own harness, both self-inflicted errors included, plus the limitations I have not solved.
• It is a product, not a demo: two packages on npm, three live pages, a CLI that works against pages I did not write, unit tests, CI with a no-LLM smoke test, Apache-2.0, and per-run JSON committed beside every report.
• The whole project was created on 31 August, inside the submission period. First commit to published packages in one day.

What I learned

• Round-trips are the mechanism on a small page; page size is the mechanism on a real one. They compound, and only measuring both makes the distinction visible.
• A tool description is a prompt. list_open_slots saying "slot ids from here are what book_slot accepts" removed a whole class of invalid calls before validation ever ran.
• Verification design is the benchmark. With a loose success regex, all five wrong-product runs would have scored as passes and I would have published "100% success, 12.9x", understating a real problem while sounding more confident.

What's next for AgentPerf

• Third-party WebMCP pages. The current tools were written by the person who wrote the tasks; that is the honest limit of these numbers, and the harness already works on any page.
• More models and providers, since --base-url makes every OpenAI-compatible endpoint a data point.
• Priced tools (x402). Every useAgentAction already carries an inert price field. When settlement is generally available, AgentPerf tells you what your fast lane is worth and x402 lets you charge for it.

Built With

• claude
• github-actions
• node.js
• npm
• openai
• playwright
• pnpm
• react
• typescript
• vercel
• vite
• vitest
• webmcp
• zod