Living Evidence

Living Evidence uses WebMCP to turn scientific papers into agent-operable tools, enabling AI to audit provenance, rerun analyses, challenge claims, and propose human-approved updates.

Comment

Inspiration

Scientific papers are designed for people to read, but AI agents increasingly participate in literature review, evidence synthesis, and research planning. Today, an agent usually receives prose or a PDF and must reconstruct the underlying data, assumptions, and calculations itself. That creates avoidable errors and makes verification more expensive than it should be.

Executable papers brought computation closer to motivated human readers. WebMCP suggests the next step: the document itself can expose typed, bounded operations directly to a visiting agent.

That led to a simple question:

What should a scientific paper become when reading can include rerunning?

Living Evidence explores that question. Its goal is not to build a “truth machine.” It is to accelerate science by lowering the cost of inspecting, reproducing, challenging, and improving published claims—while keeping humans responsible for what enters the evidence base.

Live demo: https://living-evidence.doralemon.chatgpt.site/
Source code: https://github.com/microckey/living-evidence
Demo video: https://youtu.be/fjuumUvfH1E

What it does

Living Evidence combines a human-readable scientific document with structured evidence, deterministic analysis code, declarative claim rules, and typed WebMCP tools.

The main exemplar reconstructs a historical meta-analysis of the Pygmalion effect using 19 effect-size records representing 18 experiments. Its 15 WebMCP tools let an agent:

  • inspect the data manifest and provenance;
  • rerun fixed- and random-effects models;
  • compose sensitivity analyses by excluding selected records;
  • run subgroup, moderator, cumulative, and publication-bias diagnostics;
  • inspect and execute registered claim rules;
  • render new figures into the page;
  • inspect the audit trail and create reproducibility receipts.

The results appear in the same document the human is reading. There is no need to scrape a table, copy data into a notebook, or ask the language model to perform the statistics itself.

The first screen offers a 30-second cross-examination: run the headline's rule, inspect source gaps, or copy a prompt for a real WebMCP agent. Human button clicks and tool-client calls are attributed separately, and native calls update the same visible result. The workspace includes an explicitly synthetic three-record learning sample, staged behind individual human approval and isolated from existing work.

Claims are reported as registered rule passed, registered rule failed, or inconclusive. These labels describe the result of an author-defined rule. They are explicitly not judgments of scientific truth, validity, risk of bias, or evidence quality.

The authoring workspace exposes 18 tools and accepts a narrowly defined aggregate-SMD evidence package from CSV, JSON, Quarto, or Jupyter. Every proposed record must include its source, locator, original quotation, derivation, study design, outcome, timepoint, experiment identifier, estimand, and explicit risk-of-bias status.

Agents may propose new evidence, but they cannot approve it. Only a human can accept or reject each record through the visible review interface. Approval means inclusion in that local evidence base—not independent verification of the source.

The exemplar and workspace record scientific actions in reload-persistent SHA-256 audit chains. Export produces a self-contained HTML document that works offline, plus cryptographic receipts covering the scientific state, runtime, audit prefix, and exact exported bytes.

The project also includes:

  • a 10-tool Atlas that computes coverage leads and missingness within the collected corpus;
  • an 11-tool experimental Evidence Board for mapping claims, evidence, mechanisms, and open questions;
  • a frozen PDF-versus-WebMCP evaluation protocol and local scorer.

The Evidence Board is deliberately labeled as an unverified conversation-ingestion appendix. Its Japanese source excerpts remain verbatim beside separately labeled English translations.

The benchmark currently contains zero completed comparisons, so Living Evidence makes no claim that WebMCP is already faster or more accurate.

How we built it

The scientific core is dependency-free browser JavaScript. It implements fixed/common-effects analysis, DerSimonian–Laird and REML random-effects models, subgroup analysis, one-moderator meta-regression, leave-one-record-out checks, cumulative analysis, and classical Egger diagnostics.

Statistical results are rendered as tables and SVG forest, funnel, and moderator plots inside the document. The language model selects and combines bounded operations, but the deterministic page code performs the calculations.

Registered claim rules are declarative expression trees rather than executable JavaScript supplied by an agent. This makes each rule inspectable, portable, and testable. WebMCP tools are registered from the top-level page using closed input schemas, bounded parameters, and plain structured results.

The workspace validates imported packages atomically. An invalid package cannot partially modify the evidence base, and every valid row is still staged for an individual human decision. Approved changes create a new evidence version and mark earlier claim results as stale.

For integrity, the project uses canonical JSON, SHA-256, and Web Crypto ECDSA P-256. A self-contained export carries a signed receipt for its scientific state and runtime, while a detached receipt covers the exact final HTML bytes. A command-line verifier recomputes the hashes and checks both signatures and their linkage.

The Sites deployment is a thin Vinext and Cloudflare Worker routing layer over the same static scientific documents. It avoids iframes so that WebMCP registration remains attached to the top-level document.

Unit and real-browser tests cover the exemplar, workspace, Atlas, and Evidence Board, including hostile imports, approval gates, actor attribution, persistence, corrupt snapshots, exact-byte export, receipt verification, offline file:// operation, and zero-network self-containment.

A separate, dependency-free native integration harness uses Chrome's DevTools WebMCP domain—not a mock registry or direct handler calls. In Chrome 152 it discovers the four tool surfaces, checks their schemas, invokes each overview, and verifies that a native headline-rule call changes both the visible badge and the agent ledger. Invalid claim input is rejected without ledger mutation. This is browser-client integration testing, not a claim about autonomous reasoning; ChatGPT's in-app agent remains untested.

Selected numerical outputs reproduce checked R metafor reference values to the tested precision. That verifies a narrow implementation of the arithmetic—not the transcription, model assumptions, or scientific conclusion.

Challenges we ran into

The hardest challenge was defining what every operation does—and does not—mean.

A deterministic calculation can be reproducible while using an inappropriate model. A registered rule can pass while its claim is false. A human can approve an extraction without checking the primary report. A cryptographic hash can detect changed bytes without proving authorship or long-term preservation.

Those distinctions had to become machine-readable states rather than footnotes.

The historical dataset also contains 19 effect-size records from only 18 experiments because two records represent conditions from the same experiment. The current row-wise model does not represent their covariance, so uncertainty may be understated. The project preserves the historical calculation but exposes this limitation in the manifest, analysis results, and interface. New imports reject repeated experiment identifiers until covariance-aware methods are available.

Provenance was another major constraint. The current exemplar openly reports:

  • primary reports checked: 0/19;
  • effect-size derivations independently checked: 0/19;
  • structured risk-of-bias assessments supplied: 0/19.

Representing missing verification honestly proved as important as representing the available numbers.

Integrity introduced a technical circularity: a final HTML file cannot straightforwardly contain a hash of its own final bytes. The solution was to separate the embedded scientific-state receipt from a detached exact-artifact receipt.

Deployment also required care. The build system initially treated the root scientific document as a conventional application entry and transformed its module boundary. Serving the validated documents as raw text through a thin route adapter preserved the original runtime and top-level WebMCP registration.

Native testing also caught a subtle Chrome compatibility issue: one non-ASCII schema description registered successfully but disappeared from the DevTools discovery output. An isolated reproduction identified the encoding boundary; an ASCII description preserved the schema without changing the calculations. That reinforced why testing handlers alone is not enough.

Accomplishments that we're proud of

The project’s proudest feature is that the document can lose an argument against its own registered rules.

The exemplar’s famous headline claim fails its registered rule on the full corpus, and the page reports that result plainly—without turning “rule failed” into “the claim is false.”

Living Evidence is also a working public implementation rather than a scripted mockup. An agent can compose a sensitivity analysis the author never prebuilt as a button, receive a deterministic result, render the new figure into the shared page, and leave an auditable record of the operation.

Other accomplishments include:

  • a concrete human approval boundary for agent-proposed evidence;
  • strict, inert imports from common research-authoring formats;
  • a self-contained offline scientific document;
  • signed state and exact-byte reproducibility receipts;
  • explicit provenance and risk-of-bias missingness;
  • an evaluation protocol built before making a performance claim.

Most importantly, the project treats uncertainty and incompleteness as first-class data rather than hiding them behind a polished summary.

What we learned

The most valuable scientific agent interface may not be the one with the most tools. It may be the one with the clearest epistemic boundaries.

Typed operations help, but typed limitations matter just as much. “Not assessed,” “unverified,” “outside this corpus,” and “no runs recorded” are useful outputs when they are structured, visible, and available to both the human and the agent.

Human oversight also works best as a concrete state transition rather than a general promise. Separating proposal from approval makes responsibility inspectable: the agent provides leverage, while the human controls admission into the evidence base.

The useful unit is not simply an answer generated by an agent. It is an inspectable operation whose inputs, rule, result, scope, and human decision remain visible together.

WebMCP’s strongest near-term contribution to science may therefore be reducing friction around deterministic checking—not replacing scientific judgment.

What's next for Living Evidence

The next product milestone is to connect today's separate surfaces into one versioned research project: question → source inbox → reviewed evidence → analysis → exploratory follow-up → frozen release → next iteration. The current live demo does not yet implement this end-to-end workflow, and its workspace and Atlas do not share a live evidence base.

Local development has started on the project-store and migration foundation (M3.1a/M3.1b). That work is not part of the public submission build or the demo video. The design calls for preserving complete evidence and provenance records, checking legacy snapshots against the existing restore invariants, identifying repeated imports by exact-byte SHA-256 hashes, quarantining corrupt state without deleting its only copy, and refusing writes when exclusive ownership or persistence cannot be confirmed. The migration fixes are still undergoing verification; this is a development direction, not a new shipped capability.

The planned source inbox will accept incomplete paper references without treating them as analysis-ready evidence. Numerical admission will remain a separate, human-reviewed step. Findings and follow-up hypotheses should link to the exact analysis and evidence revision that generated them; a hypothesis suggested by the same data must remain exploratory, not be relabeled independent confirmation. The experimental Board will remain outside numerical evidence admission.

The planned release workflow will freeze a research snapshot and fork the next iteration from it. Downloading a package will not be described as publishing online: automated hosting, durable public archives, and externally anchored identity remain separate future work.

Alongside this product work, we plan a preregistered comparison of PDF-based and WebMCP-based workflows across independently authored documents, agent models, and human reviewers. The evaluation will measure task accuracy, completion time, calculation errors, provenance recovery, and human correction effort before making any performance claim.

Scientifically, the next steps are to:

  • verify the exemplar against primary reports;
  • add instrument-specific risk-of-bias workflows;
  • support dependent effects through multilevel or robust-variance methods;
  • test generalization beyond one aggregate-SMD corpus.

For publishing, Living Evidence needs immutable versions, durable archives, externally anchored author keys, and receipts connected to persistent identifiers.

For authors, the goal is to make Living Evidence a routine export from tools such as Quarto, Jupyter, metafor, and structured review databases—not a separate engineering project.

The Evidence Board will remain experimental until its ingestion, citation, and verification pipeline has been independently evaluated.

Longer term, Living Evidence could become an open layer through which scientific documents expose not only what they claim, but exactly how those claims can be inspected and challenged.

For an agent, reading can be rerunning.

Built With

Share this project:

Updates