Runtime Truth

Inspiration

As AI agents move from reading websites to taking real actions — booking, buying, submitting forms, and changing application state — a new reliability question appears:

How do we know the action actually happened?

A tool can return SUCCESS, but the authoritative application state may tell a different story.

Runtime Truth explores that gap.

What it does

Runtime Truth exposes a WebMCP tool called reserve_table.

In the demo, an agent requests a reservation for two people at 7 PM.

The tool reports:

SUCCESS — Reservation ID: R-1042

But the reservation is deliberately not persisted.

Runtime Truth then independently checks the authoritative application state.

It finds:

RESERVATION NOT FOUND

and flags:

DIVERGENCE DETECTED

The core principle is simple:

Don’t trust the tool response. Verify the state.

How I built it

The project uses WebMCP's imperative API to register a structured website tool that agents can discover and execute.

The execution flow is:

1. The WebMCP tool performs the requested action.
2. The application receives a declared result.
3. Runtime Truth independently queries the authoritative post-action state.
4. The declared result and observed state are compared.
5. A mismatch is surfaced as execution-state divergence.

The demo uses deterministic fault injection so the failure can be reproduced consistently.

The application is built with Python, Flask, JavaScript, and WebMCP, and deployed on Render.

Challenges

The most important challenge was separating tool execution from verification.

If verification simply trusts the same response that declares success, it cannot detect false-success states.

I also had to test WebMCP registration, discovery, and execution independently. The final implementation was verified using Chrome's WebMCP testing support:

• tool registration
• tool discovery
• WebMCP tool execution
• independent post-action state verification

What I learned

A successful tool response is a claim about execution, not necessarily proof of the resulting state.

As WebMCP makes websites increasingly agent-native, verification may become especially important for consequential actions such as purchases, reservations, submissions, and workflow changes.

I also came across Greg Isenberg's discussion of an "agent mystery shopper" concept for testing whether agents can actually complete important website journeys.

Runtime Truth approaches a narrower technical version of that problem:

When a tool declares success, did the expected state transition actually occur?

I see that discussion as an independent signal that completion verification is becoming relevant as agents perform more real-world actions on the web.

What's next

Runtime Truth is intentionally small.

The next step would be testing the same verification pattern across additional state-changing actions such as purchases, submissions, and asynchronous workflows.

For this challenge, the goal is narrower:

demonstrate clearly that tool success and state truth are not always the same thing.

Built With

• flask
• github
• html
• javascript
• python
• render
• restapi
• webmcp