Actionwitness

AI agents say “done.” ActionWitness proves it—checking WebMCP actions against real outcomes and turning failures into tamper-evident, replayable tests.

Comment

Inspiration

Most WebMCP evaluation ask whether an agent called the right tool with valid arguments. That matters, but it stops at the tool's own response without evidence.

Consider an agent calling apply_discount({code: "SAVE20"}). The tool returns {"status": "success", "total": "20.00"} and the call-level evaluator passes it. But the shopper's cart still totals 25.00. The call was well formed; the business outcome was wrong.

This is a silent success. Nothing crashes, the logs look healthy, the agent moves on, and the user is given false confidence.

I built ActionWitness to close that gap. The name is the thesis:

The agent said it did the thing. ActionWitness shows what actually happened.

ActionWitness complements call-level evaluators rather than replacing them. A call-level evaluator checks whether the model used the tool correctly. ActionWitness checks whether the tool produced the promised real-world outcome. Strong assurance needs both.

What it does

ActionWitness is a target-neutral assurance harness for WebMCP applications. It records what a tool claimed, independently observes authoritative business state, and evaluates both against a human-defined outcome contract.

The core mechanism deliberately keeps two channels separate:

Agent intent
    |
    +--> WebMCP tool call --------> tool-reported result (the claim)
    |
    +--> independent observer ----> authoritative business state (the outcome)
                                             |
Human-defined contract ----------------------+--> deterministic verdict

A successful tool response is evidence, never manufactured observed state. If the authoritative state is unavailable or the evidence chain cannot be verified, the result is an explicit non-pass—not a guessed success.

ActionWitness provides:

  • Outcome contracts. A human declares the required business outcome. A deterministic, target-neutral kernel produces the verdict; an LLM never acts as the authoritative judge.
  • Independent observation. Tool output and observed state are stored as different evidence types with explicit source classification.
  • Hash-linked evidence. Evidence is append-only, canonically serialized, content-addressed, and verified before it is trusted.
  • Human confirmation. Consequential mutations require a server-issued confirmation bound to the workspace, run, action, arguments, and expiry. An agent cannot create, broaden, or approve its own consent.
  • Fail-closed idempotency. Retries reuse a key only for identical intent. Reusing a key with changed intent is refused.
  • Regression replay. A failed journey can produce a self-contained case that restores its fixture and reproduces the same classification locally or in CI.
  • Repeated trials and correlation. Frozen intent variants run in isolated workspaces and correlate with imported Google webmcp-evals reports. Trials without a valid observation are excluded with a reason rather than counted as passes.
  • Human-frozen intent variants. An optional Gemini backend drafts rephrasings, but generated rows arrive unapproved. Drafting and freezing use separate endpoints, so one call can never generate and approve an intent.

The number a call-level evaluator cannot produce

ActionWitness presents a simple disagreement matrix:

Observed outcome passed Observed outcome failed
Call-level evaluator passed Supported success Silent outcome failure
Call-level evaluator failed Outcome succeeded despite call issue Both layers failed

The key measure is straightforward: among the trials your call-level evaluator passed, how many still failed in observed business state? In our recorded demo suite, that silent outcome failure rate is 0.5000. The UI names the disagreement cell in words rather than asking a reviewer to infer it from raw counts.

Three proofs, not one

  1. A deterministic false-success target. The Buggy Store has pre_fix and post_fix modes. The same journey fails or passes with exactly one declared fault profile changed, making the evidence a controlled comparison rather than a staged assertion.

  2. An authorized Shopify development store. The integration is intentionally narrow: one operator-authorized development store, one server-controlled variant and currency, and cart-only behavior. Shopify's native WebMCP tools perform the agent-side actions. A reviewed theme bridge observes the shopper's own cart through Shopify's locale-aware, same-session /cart.js API—a channel independent of the tools' self-reports. The proof has been exercised end to end against the configured development store; checkout, orders, payments, bulk operations, and arbitrary remote targets remain out of scope.

  3. ActionWitness verifies itself. A separate self_target distribution drives ActionWitness through its public API and observes a different server-minted workspace through the same protocols used by third-party integrations. It declares only the target-neutral core and an HTTP client, so it cannot import the repository, service, or database. Pointing observation at the recording workspace instead makes the run fail with maximum_mutations: the product catches its own loss of observation independence.

How I built it

Python 3.12 owns contracts, orchestration, persistence, observation, evidence, evaluation, replay, and CLI behavior. FastAPI and Pydantic validate the service boundary. SQLite is the durable source of truth, evolved through ten explicit, tested migrations.

React with strict TypeScript stays a minimal interaction and browser-integration layer; it does not duplicate Python-owned verdict or consent logic.

All direct WebMCP access is isolated in frontend/src/webmcp/adapter.ts. The adapter resolves document.modelContext first and navigator.modelContext as a compatibility fallback, normalizes errors, propagates cancellation, and performs StrictMode-safe cleanup. The bundled harness and Buggy Store share one origin, while the authorized Shopify proof uses an explicitly configured external store and browser-session observer.

ActionWitness publishes 17 lifecycle-scoped WebMCP tools, including seven read-only inspection tools for timelines, comparisons, regression cases, audit reports, and benchmark summaries. Only the tools valid for the current workspace phase are registered. The deliberate exclusions are just as important: an agent cannot approve consent, authorize an audit, submit its own audit evidence, freeze intent variants, or curate benchmark results.

The repository's safety rules are executable gates rather than documentation alone. The current suite contains:

  • 3,277 Python tests, including 227 architecture checks
  • 433 frontend tests
  • 79 Playwright end-to-end tests covering the real service, a conformant WebMCP boundary, the adapter, independent observation, and persisted evidence

Isolation scripts also create fresh virtual environments and install the core and demo store distributions independently. This verifies that package boundaries work when unrelated workspace dependencies are absent.

Challenges we ran into

Independent observation is not always available

The product only works if the observation channel is meaningfully independent of the tool response. Parsing a tool-controlled message or treating a success payload as observed state would collapse the distinction ActionWitness exists to enforce.

For the authorized Shopify development store, the independent channel is the shopper's same-session Cart API. A server-side request would create a different session and observe the wrong cart, so the bounded observation runs inside the operator's browser session and sends only the required cart evidence back to the harness. If no acceptable channel exists, ActionWitness returns observation_unavailable instead of guessing.

Agents can operate the harness, but cannot grade themselves

WebMCP lets an agent arm a run, exercise a target, and inspect findings. That power needed a hard boundary: humans define correctness and authorize consequential actions; agents exercise tools and inspect evidence.

Arch enforces that separation in schemas, routes, and server-issued confirmation—not in prompt wording. No published tool can approve consent, authorize an external audit, submit the observation transcript under audit, or author its own passing condition.

A consequential action pauses for a human decision bound to the exact request

Self-witnessing can accidentally become self-observation

Letting ActionWitness target itself introduced a subtle circularity: the workspace recording the audit cannot also be the supposedly independent workspace being observed.

The server therefore mints and binds a distinct observed workspace before execution. Published tools do not accept an agent-supplied workspace override, unknown fields are refused, and an unbound adapter fails closed. This turned observation independence from a convention into a structural property.

WebMCP is still evolving

The browser API and host implementations are changing. We isolated the entire direct API surface behind one adapter and kept the full human workflow functional when WebMCP is unavailable. Browser support is reported as a fact, never treated as permission to weaken the workflow.

Accomplishments that we're proud of

  • The contradiction is visible on one screen: the tool reports success, the independently observed cart remains unchanged, and the run fails as false_success_or_state_mismatch while the call and trajectory layers still pass.
  • The Shopify development-store proof works end to end: native Shopify WebMCP actions on the agent side, same-session cart observation on the witness side, and no shopper credential stored by ActionWitness.
  • The harness passes its own contract through /api/v1: arm a self contract, drive two published tools, and receive a deterministic verdict. Breaking workspace independence makes it fail for the right reason.
  • Failures become reproducible assets: a regression case restores the fixture and reproduces the classification locally and in CI.
  • Call-level and outcome-level evaluation work together: imported webmcp-evals results are correlated rather than reimplemented or replaced.
  • Adversarial cases have direct coverage: evidence tampering, duplicate mutations, report corruption, stale approval, source disagreement, cancellation, and unavailable observation all fail explicitly.

A failed journey becomes a deterministic regression case that can be replayed

What we learned

A tool's self-report is evidence, never proof. Taking that sentence literally shaped the storage model, API boundaries, observer protocols, verdict engine, and demo.

Determinism is a product capability. Injected clocks, identifiers, and randomness are what let a regression reproduce the same classification rather than merely resemble the original failure.

An unavailable witness must say it could not see. observation_unavailable is more trustworthy than a green verdict constructed from a non-independent source.

Human authority must be structural. If an agent is prohibited from approving consent or grading itself, the capability must be absent from its tool surface—not merely discouraged in a prompt.

Architecture rules become durable when they are executable. Tests that reject target-specific imports in the core or WebMCP access outside the adapter keep those boundaries true as the product grows.

What's next for ActionWitness

  • Publish the existing regression-case schema as a stable interoperability specification.
  • Make contract authoring simpler while keeping final correctness and approval human-owned.
  • Package an observation-provider SDK so teams can add target adapters through the existing public protocols.
  • Expand browser and accessibility validation as WebMCP stabilizes.

ActionWitness is deliberately not a general-purpose agent evaluator. It owns one missing layer: standing outside a tool's account of events and verifying what actually happened.

Built With

Share this project:

Updates

Submission history