-
-
An autopsy for dead scientific hypotheses. It takes a claim and returns Confirmed, Revoked, or Needs data.
-
Case 01, PMID 19076952. The claimed window doubles every 2.4 years; extended to 2026, every 14. The exponential did not last.
-
Assumption ledger: ambiguous phrases from the paper are quoted, given an id, and resolved in the open — never silently.
-
Case 02, PMID 37314101. The 95% interval contains zero, so the verdict is Needs data, not Revoked.
-
Six stages from a PubMed ID to a verdict. Each stage carries the requirement it implements as an annotation.
-
Specs, steering and the spec-coverage hook live in /.kiro at the repo root. The method is committed, not described.
-
Provenance: every reported number maps to the requirement it serves and to the line of code that produced it.
Inspiration
Every year tens of thousands of papers state a hypothesis, report a p-value, and are never revisited. The claim gets cited, the citation reaches a review, the review reaches a guideline, and nobody asks whether the trend still holds. What makes that strange is that for a whole class of claims the data needed to re-test them is now public and machine-readable — PubMed alone will tell you how many papers in a field were published in any year back to 1950.
The bottleneck was never the data. Re-running an old analysis by hand means reading a methods section full of ambiguity, making a dozen silent judgment calls, and rewriting the statistics from scratch. Weeks of skilled labour per claim, and no record afterwards of what you assumed.
Spec-driven development supplied the missing piece. If a paper's claim is written as testable requirements before any code runs, the ambiguity becomes a visible artifact instead of a hidden one, and the pipeline that follows is auditable number by number.
What it does
Necropsy is an autopsy for scientific claims. Give it a PubMed ID and it returns one of three verdicts: Confirmed, Revoked, or Needs data.
Six stages:
- Fetch — abstract and MeSH terms from NCBI E-utilities. Every request, URL, timestamp and returned count is written to an access log.
- Extract — the trend claim is located as a verbatim span of the abstract by a deterministic extractor, not an LLM. The claim, its quoted window and its subject phrase become structured records.
- Resolve — the phrase the paper used is turned into a query PubMed can answer. Where it can be read two ways, two operationalisations are built and both are run; the report says whether they agree.
- Count — one E-utilities request per publication year per query variant.
- Fit — Poisson log-linear regression in R, with Pearson dispersion measured and a quasi-Poisson interval computed before any verdict is assigned.
- Decide — a pure function, tested by truth table, assigns the verdict.
Case one, PMID 19076952 (Robotic surgery, 2009), claims exponential growth over the preceding decade. On the claimed window 1999–2009 the log-slope is 0.289/yr, doubling time about 2.4 years: Confirmed. Extending the window to 2026 the slope falls to 0.049/yr, doubling time about 14 years. The claim was true; the exponential did not last.
Case two, PMID 37314101 (Cryptic Species in Ecotoxicology, 2023), returns Needs data. Its 95% interval runs from −0.16 to +0.33 and contains zero. The tool refuses to call that a refutation.
Every report carries a provenance table (each number → requirement → line of code), an assumption ledger (each ambiguous phrase, quoted, and how it was resolved), and a fragility line naming the one assumption that could flip the verdict.
How we built it
Kiro was the entire development environment, and the evidence sits in /.kiro at the repository root rather than in this form.
Specs. .kiro/specs/paper-autopsy/ holds requirements.md (ten EARS statements, A-001 to A-010), design.md and tasks.md. A-006, the verdict rule, was first written loosely enough that an interval containing zero produced "Revoked". The second case exposed it: we changed the requirement, then the tests, then the code. That order is the point of the method — and it is why A-006 now says in so many words that an unresolved interval is not a refutation.
Steering. product.md fixes what the tool refuses to do. statistical-conventions.md fixes effect reporting — estimate with a 95% CI, never a p-value alone — plus the missing-data and multiple-comparison policy. tech.md fixes the language boundary: statistics live in R, so a reviewer can read the model call the way they read a methods section. Two visible consequences in the output — every number carries an interval, and "Needs data" is never rounded up into "Revoked" — come straight out of that steering file.
Hooks. .kiro/hooks/spec-coverage.json fires when a source file is edited and asks the agent to verify that every block producing a number in the report carries a # REQ: A-### annotation pointing at an id that exists in requirements.md. An unannotated block means the task is not complete. The provenance table is trustworthy because coverage is enforced while writing, not audited afterwards.
Python does retrieval, extraction and reporting. R does the model. Streamlit serves the web demo.
Challenges we ran into
Ambiguity is where claims die. "Publications concerning robotic-assisted laparoscopic surgery" is not a query. Turning it into one, and writing down that we did, is three of the four entries in the assumption ledger.
Queries lie. Our first MeSH formulation OR-ed two descriptors and returned 39,540 records; AND-ing them returned 1,335. Neither is "correct", which is why the report prints the query it actually ran and reports whether the two operationalisations agree.
The verdict rule was the hardest thing to make honest. The temptation is to read "the data no longer show the effect" as a refutation. Usually it is a wide interval, which settles nothing. Moving that case to Needs data cost us a stricter system and bought a defensible one.
Three points are not a trend. A window with fewer than five publication years is refused a verdict outright.
What we're proud of
A report where every number traces to a written requirement and a line of code, ending in a single sentence that says what would overturn the verdict.
What we learned
Requirements are not bureaucracy; they are what makes an AI-assisted analysis defensible. Naming an assumption costs nothing. Discovering one during review costs a rewrite.
What's next
- More claim families — association and difference claims, not only trends
- Batch mode over a paper's whole reference list
- Exporting the access log as a draft Methods section
Honest scope
Retrieval calls E-utilities directly from Python: there is no MCP server, no custom agent, and testing is unit tests plus golden cases rather than property-based testing. Only abstract-level bibliometric trend claims are supported today; any other claim type returns Needs data by design. The repository's status table lists exactly which requirements are implemented and which are not.