Thurstone by Invarra

• 

  Thurstone catches a successful-looking native action that violates the owner's contract.

• 

  Correct WebMCP code does not guarantee that an agent chooses the right action.

• 

  Judge Quick Start tests a baseline, a planted site fault, and a live semantic collision.

• 

  The expected outcomes are declared before the fresh agent sees the live catalog.

• 

  The owner begins by separating read-only review from explicit checkout authorization.

• 

  Thurstone ingests the current WebMCP catalog and lets the owner select real tools to test.

• 

  Agent-visible titles and descriptions remain editable while real handlers and schemas stay fixed.

• 

  Representative requests become repeatable contract cases with schema-derived arguments.

• 

  One private expiring command hands the isolated suite to a fresh ChatGPT Browser task.

• 

  The owner page tracks independently verified results without exposing the private command.

• 

  Without a semantic release check, a correct handler can still ship the wrong behavior.

• 

  Define, arm, test, verify, diagnose, save, and rerun: one repeatable release loop.

• 

  Reference evidence keeps semantic behavior and Invocation Integrity in separate scores.

• 

  Meaning is the unit; live WebMCP behavior is the evidence.

• 

  Thurstone turns controlled semantic variation into contracts, observed actions, and verified effects.

Video quality: For readable interface text, select 1080p in the YouTube player settings; YouTube may default to a lower resolution.

Inspiration

I built Thurstone because WebMCP lets agents act on a website through custom tools, but nothing ensures they choose the right one.

A tool can have a valid schema and a handler that works correctly in isolation, yet an agent can still choose the wrong action for what a person meant, supply the wrong arguments, or cause an effect the website owner prohibited. The code worked, but the behavior was wrong.

That happened in a ChatGPT regression run. The request was “Show me my current order.” The contract required order_review, but the agent selected cart_get. Nothing crashed, and the site remained unchanged, but the action didn't match the declared meaning. Thurstone flagged it and recommended fixing the boundary between cart contents and a complete order review.

Thurstone is a pre-release testing and audit system for that boundary: what a person meant, what the agent chose, and what the website actually did.

WebMCP supplies a site-owned tool catalog that an agent can discover and invoke. It connects natural language, deployed tool definitions, browser-native invocations, canonical arguments, and real site effects in one auditable chain. Without WebMCP, Thurstone could test only a detached API or simulated action.

What it does

The website owner selects tools from the WebMCP catalog, writes requests as a visitor might express them, and declares the expected tool, argument constraints, allowed and prohibited effects, replay policy, and trusted state source.

A supported external agent receives only the request and selected agent-visible catalog. Expected actions, effects, and assertions remain hidden. The agent uses the page’s native WebMCP tools; Thurstone admits one action per test, captures the invocation and arguments, and checks them against independent before-and-after website state and an append-only operation ledger.

Each executed test returns PASS, ISSUE, INCOMPLETE, or UNAVAILABLE. NOT RUN marks a queued case that was never executed—for example, because a stateful journey stopped after an earlier issue. Results show what was expected, what the agent did, what changed, what diverged, and where to investigate. Verified cases can be preserved as regressions.

The collaboration is direct: the owner defines what a request should mean; a visitor’s agent uses the available WebMCP tools; Thurstone checks whether the action and resulting website state match that meaning. The human remains responsible for the contract, release decision, and approval of actions such as checkout.

For owners, semantic failures become reproducible release tests instead of customer reports. For visitors, agents are more likely to do what the person meant—and to ask when a request is insufficient for action. Any store, publisher, SaaS team, or platform exposing WebMCP tools can test this boundary.

Judge Quick Start: open thurstone.invarra.ai/judge, press Arm, and copy the generated command. Paste the complete @Browser Open command into a fresh GPT-5.6 Sol or Terra ChatGPT Desktop chat. Do not use Chrome’s side panel. Follow the page controls; results open automatically. The controlled example also runs in WebMCP-enabled Chrome.

Full Demo: open thurstone.invarra.ai/demo, choose one to four live tools, build schema-derived cases, and run a regression suite or stateful journey. Send its private expiring command to a fresh ChatGPT Desktop chat, then inspect, export, or rerun the results.

How we built it

The reference catalog contains cart_get, cart_update, order_review, checkout_request, and a state-dependent checkout_cancel. The site registers its tools with document.modelContext.registerTool(). A shared deterministic reducer, native handler traces, browser-local site-owned state, and an append-only ledger produce trusted before-and-after evidence independently of the tool’s response.

Contract authoring and agent execution are separated. The agent command contains the authorized requests but not the expected actions or effects. A short-lived Redis ledger coordinates handoff, claim, start, case admission, resolution, and owner synchronization. Only one eligible agent action can count for each case, and the ledger contains no checkout data or customer account information.

Regression suites run every case in one agent chat, resetting the fixture between cases and continuing after issues. Stateful journeys keep one agent context, carry verified state forward, and stop when an issue makes later steps unreliable.

Chrome’s WebMCP eval guidance recommends probabilistic agent evaluations alongside deterministic application tests. Thurstone binds those layers into one live owner-facing contract, adds prohibited effects, and verifies the resulting site state.

Thurstone development began August 26, 2026. Its application, WebMCP catalog, contract builder, agent path, native-call capture, effect verification, diagnostics, evidence pipeline, Demo, and judge experience were all created during the challenge, as the repository history shows. Two Invarra measurement notes—LIP and CSR, published June 28—provided only conceptual background. No Thurstone software or agent workflow existed beforehand.

Challenges we ran into

1. Different consumers format arguments differently. One passed arguments as an object; another used a JSON string. Thurstone calibrates once with a read-only call, normalizes the format without hiding real argument errors, preserves cancellation, and never retries a mutation with a different encoding.

2. One test needs one accountable action. If several tools can count, the result becomes ambiguous. Thurstone admits only the first eligible invocation. Longer workflows become ordered journeys or independent regression cases.

3. Tool output is not proof of the outcome. One test admitted the expected cart_update, but the required state change never appeared. Tool output is therefore corroborating evidence; canonical arguments, trusted before-and-after state, and the ledger determine the verdict.

4. The agent must not receive the answer key. A single-use, expiring handoff contains only the authorized request and selected live catalog. The expected action and effects remain hidden until the test finishes.

Accomplishments that we're proud of

Three real Thurstone runs cover 11 tests: 8 passes, 3 issues, 0 not run. Two are ChatGPT-agent runs; the third is a disclosed no-model control. An ISSUE means the declared agent/site contract diverged—not that Thurstone failed.

• Regression suite — 3 tests: 1 PASS, 2 ISSUES, 0 NOT RUN: Thurstone caught a missing required cart effect and an agent selecting cart_get instead of order_review.
• Continuous journey — 7 tests: 7 PASS, 0 ISSUES, 0 NOT RUN: seven native calls in one ChatGPT context; checkout transitioned 2 → 3, mug quantity 3, one pending approval, $79 total, and no unmodeled change.
• Controlled mismatch — 1 test: 0 PASS, 1 ISSUE, 0 NOT RUN: order_review was invoked where checkout_request was required; canonical arguments and the required checkout effect diverged.

The separate 24/24 reference regression is bounded coverage—not an independent benchmark.

Chrome 152 compatibility: a native checkout passed all seven assertions covering the contract, arguments, before-and-after state, and ledger.

What we learned

A correct schema does not guarantee that an agent understood the user. A handler’s success message is not proof; trusted site state and ledger evidence are. Clarification, no action, and incomplete runs must remain honest outcomes. Thurstone can identify the exact contract assertion that diverged and suggest where to investigate, but it does not claim to know the model’s private reasoning.

What's next for Thurstone by Invarra

The challenge build uses a fictional checkout and preconfigured tools; it does not yet connect to arbitrary sites. The next step is to extend the same contract and effect-verification workflow to owners’ own WebMCP catalogs and test environments.

Today, semantic tests require ChatGPT Desktop’s in-app Browser, while WebMCP-enabled Chrome supports native compatibility checks. Thurstone remains a testing and audit system—not runtime enforcement or certification—and every result applies only to the declared contract, agent, fixture, and tested build. All checkout activity is simulated.

Built With

• chatgpt
• chrome
• gpt-5.6-terra
• json-schema
• next.js
• openai
• playwright
• react
• typescript
• upstash-redis
• vercel
• vitest
• webmcp
• zod