Maxwell StudyClaim Gate

A two-axis check on whether a study earned its claim—comparing the paper against the trial's registry history to catch outcome switches. Built with Codex + GPT-5.6.

Comment

Inspiration

Before a scientific claim is published, approved, or acted on, someone has to verify that the evidence earned it. Medical editors, research-integrity teams, scientific communicators, and regulatory and compliance reviewers do that verification manually today — slow, specialized work that is hard to do consistently. A claim can accurately restate a paper and still rest on a study whose reported primary outcome doesn't match what was prospectively registered — a discrepancy a single read rarely catches.

Published cohorts bear this out: Chan et al. (JAMA, 2004) found major primary-outcome discrepancies in 51 of 82 trials with a defined primary outcome, none disclosed as amendments; Holst et al. found 14.3% of trials had primary-outcome changes that would be invisible when comparing only against the latest registry version, with 96.7% of outcome changes undisclosed; and Boutron et al. (JAMA, 2010) found spin in 68% of abstracts of nonsignificant trials.

What it does

Maxwell StudyClaim Gate makes that review faster, more reproducible, and easier to audit. It evaluates a medical claim along two independent axes:

Axis 1 — claim vs. study: whether the study supports the proposed claim, checking measured endpoint, design-appropriate causal language, clinical-versus-surrogate status, population, timeframe, comparator, outcome status, single-study certainty, and whether the study's reported result direction matches what the claim asserts.

Axis 2 — study vs. registry: whether the published study's primary outcome matches its prospectively registered outcome history, including the registry's version history.

Keeping the axes separate is the point. A claim can faithfully restate a paper — and pass Axis 1 — while the paper's primary-outcome hierarchy no longer matches what was originally registered, and therefore fail to clear Axis 2. The interface always shows both findings; the combined state never erases either one. It states facts and dates, routes unresolved concerns to methodologist review, and does not allege intent.

The demonstrated case: the replayed hero is a real published randomized trial (PMC4057030, registered as NCT01401283). The claim that perioperative goal-directed therapy reduces postoperative complications matches the paper, so Axis 1 passes. But the registry history shows a separate fact: as originally registered, Hospital Stay was the primary outcome and Postoperative Complications was secondary. The paper published in 2013. Roughly two and a half years later, in a registry update submitted in 2016, those roles were swapped. The originally-registered primary outcome — hospital stay — was null (11 vs. 10 days, p=0.929). The result: Axis 1 PASS, Axis 2 REGISTRY_OUTCOME_SWITCHED, combined state METHODOLOGIST_REVIEW_REQUIRED — a claim that clears every ordinary check, held only by the independent registry axis.

How we built it

The build started with research, not code. Before writing the application, I ran the problem space through three different ChatGPT-5.6 retrieval methods — Deep Research, GPT-5.6 Pro, and web search — deliberately, on the assumption that each uses somewhat different retrieval and reasoning and would surface slightly different sources and framings. I then cross-examined the three for alignment and divergence, keeping the findings that held up across methods and flagging where they disagreed. That process — the same triangulate-and-check-for-agreement logic the product itself applies to evidence — worked through the literature on how medical claims break down between a trial and its publication: outcome switching, registry version histories, surrogate endpoints, and reporting spin.

The Centre for Evidence-Based Medicine's COMPare project was a turning point: in an audit of 67 randomized trials, only 9 reported their outcomes perfectly, with 354 pre-specified outcomes left unreported and 357 new outcomes silently added across the cohort. Analyses of ClinicalTrials.gov put the discrepancy rate between prospective registration and publication at roughly 31–62% of trials. That research isolated a specific, high-value failure mode most claim-checking ignores: a trial's primary and secondary outcomes being switched in the registry after publication, so the published claim looks clean against the current registry while contradicting what was originally, prospectively registered.

I then built the evaluation corpus with a deliberate sourcing and verification methodology rather than convenient examples. Every candidate study had to have actively resolving identifiers (DOI, PMCID, NCT — a non-resolving identifier breaks the chain of custody), a strictly verified open-access license (CC0 or CC BY only, with CC BY-NC, BY-ND, BY-SA, and unlicensed author manuscripts excluded), and, for the priority category, a ClinicalTrials.gov version history showing a modified primary outcome. Searching the real trial record under those constraints surfaced the Salzwedel trial (NCT01401283), where the primary and secondary outcome roles were swapped in a 2016 registry update — roughly two and a half years after the 2013 publication. The product was designed backward from that real, reproducible case: the two-axis structure exists because a single-axis check, claim versus paper, structurally cannot catch it.

The system was then built with OpenAI Codex, block by block, with executable tests at each stage — schema and canonical document, quote-anchored source verification, the Axis 1 rules, registry normalization and version diffing, aggregation, typed weakening, the replay interface, and a tamper-evident SHA-256 receipt. Codex generated the schemas, the rule functions with their positive and counterexample tests, the registry version-diff logic, the receipt binding, and the replay UI; I set and held the architecture boundary, the two-axis separation, the real registry role-swap hero case, and the result-direction check. 107 tests pass from a clean clone.

How it works

The technical boundary is deliberate: GPT-5.6 reads and extracts; deterministic Python decides. At runtime, the model converts unstructured study text into a strict, quote-anchored extraction and decomposes the claim into typed fields — but it never decides whether a claim passes. Quote anchoring, evidence authority, the rule checks, registry version diffing, aggregation, and receipt verification are ordinary, testable code. This separation has empirical support: in a published risk-of-bias evaluation (Huang et al., JMIR, 2025), applying a fixed decision algorithm to model-generated answers rather than relying on the model's direct judgment raised accuracy from 70% to 90%.

Every decision is bound into a tamper-evident SHA-256 receipt covering each intermediate representation and recorded decision — change one bound fact, including a single registered outcome, and verification fails. The architecture names four invariants — no-source-no-clearance, veto-only, two-axis independence, and release-gating — and the README documents exactly which are enforced today versus specified as planned, rather than implying the full design is live. GPT-5.6 handles the language-intensive reading; Codex-built deterministic code makes every decision, so the model's reading is auditable and can never unilaterally clear a claim.

Challenges we ran into

The hardest problem was keeping the model's confidence from leaking into the verdict. An early version let Axis 1 pass claims without checking whether the study's result direction actually matched what the claim asserted — a claim could clear the structural checks while the evidence pointed the other way. The fix was a three-state result-direction precondition that distinguishes a decisive supporting direction, a decisive contradiction, and a non-decisive or conflicted signal — added without breaking the hero case, whose reported magnitudes are conflicted across passages but whose explicit directional language decisively agrees on benefit.

A second challenge was honesty about scope: separating what the engine actually enforces from what it is designed to enforce, and documenting the difference plainly rather than implying the whole design is live.

Accomplishments that we're proud of

A working two-axis system that catches a real, non-obvious failure — a registry primary/secondary outcome switch submitted years after publication — that ordinary claim-checking misses entirely. An architecture where the model can never unilaterally issue a verdict, backed by 107 tests passing from a clean clone. A tamper-evident receipt where changing a single bound fact breaks verification. And a demo, visual cards, and README where every number, date, and state is verified accurate against the source — fitting for a tool built to catch claim-source mismatches.

What we learned

That moving the final judgment out of the model and into deterministic rules isn't just safer — it is measurably more accurate (the 70% to 90% precedent), and it makes every decision auditable and reproducible. And that for a verification tool, the discipline of the build has to match the discipline of the product: the demo, the evaluation labels, and the documentation all had to hold to the same claim-versus-evidence standard the tool enforces.

What's next for Maxwell StudyClaim Gate

Enforcing the planned invariants — a required-field completeness gate, a live source-backed veto verifier, and a standalone release-artifact boundary — then broadening beyond the single replayed study to live extraction across arbitrary papers, and expanding registry coverage.

Before a scientific claim is published, approved, or acted on, someone has to verify that the evidence earned it. StudyClaim Gate makes that verification faster, more reproducible, and auditable — and keeps the human in the loop for the judgment that matters.

Built With

Share this project:

Updates