Verbatim

Inspiration

Extracted clinical trial numbers flow into meta-analyses, and meta-analyses into treatment guidelines. A wrong sample size does not stay in the spreadsheet — it ends up in somebody's treatment decision.

LLM extraction made pulling those numbers out of PDFs easy, and that is exactly the problem. A model will read a trial report and tell you it randomised 240 patients — fluently, with a citation — and be wrong often enough to matter. The reviewer downstream cannot tell which numbers were read and which were composed. Every extraction tool we looked at asks you to trust the extractor. In regulated work you cannot.

What it does

Verbatim refuses to report a value on an extractor's say-so.

Every proposed field must name a page and quote a span. A deterministic checker then goes and looks in the page text Nutrient DWS actually returned. If the span is not there — character for character, after whitespace and ligature folding — the field is never rendered as data. It is rendered as a failure.

That is the whole product. The trust boundary is a substring search over a document, not a model's self-report, and that property does not weaken when the extractor gets smarter.

Around that core:

• Cross-check. Re-anchor a finished record against a different rendition of the same document — a scan, a re-typeset, the copy that was actually filed. Where the two disagree is where a wrong number enters the evidence base.
• Live verification. Anchoring proves a number is in the document. It cannot know the document was retracted last year. SerpApi answers that.
• A tamper-evident log. Every step and every human decision is one append-only line carrying the SHA-256 of the line before it. On the demo page your own browser recomputes the entire chain, and "Break a link" performs a real edit that makes it really fail.
• An evidence table that structurally cannot contain a value which failed its anchor check.

How we built it

Nutrient DWS does the document work: one POST /build call carrying an ocr action and a json-content output, so a scanned page and a digital PDF arrive downstream in exactly the same shape. That returned page text is the ground truth the anchor checker searches — nothing is ever asserted that DWS did not return. The client is standard-library only, with hand-rolled multipart, so the repo clones and runs with no pip install at all.

SerpApi does the outside-the-document work: google_scholar for existence and citation weight, and google scoped to Retraction Watch, PubPeer and PubMed for retractions and expressions of concern. Every check logs its exact query string and timestamp, so verification is auditable the same way extraction is. This is the part that cannot be faked with a better model: a retraction is a fact that cannot live inside the PDF it applies to. It arrives after the PDF was written.

The extractors are deliberately deterministic regexes rather than a model — byte-identical output on the same input, no key required, and they fail locally instead of confidently. A model backend can layer on top, held to the same anchor check.

The corpus is three real open-access randomised trials pulled from Europe PMC, one of them genuinely retracted, plus a fourth document we generated: the same paper rasterised at 150 dpi, rotated a third of a degree and JPEG-crushed until it has no text layer at all.

Challenges we ran into

Being honest about our own findings. The first cross-check run proudly reported four fields "not surviving" the scan, and the obvious headline was "four extraction errors caught". It was wrong. Two were derived labels like none declared that were never literal text, and the rest were whitespace-normalised p-values sitting right there in the document. A tool about provenance that cries wolf about missing values commits the exact sin it exists to prevent. So drift grew three severities, and the honest verdict on our own demo is now the quieter, better one: no value went missing; four quotes did.

Anchoring is provenance, not semantics. On one paper the ethics extractor confidently picked up the trial registration number sitting in the same sentence — and anchored it perfectly, because it really was in the document. An anchored wrong value is worse than no value, because it looks checked. That is why registry-shaped codes are now rejected as approval numbers, and why there is a human in the loop at all.

Reading order. OCR on a two-column journal page returns text blocks in no useful order and shreds every sentence. Sorting observations into columns by bounding box before reading top-to-bottom took the scanned rendition from unusable to within 14 characters of the digital text.

Accomplishments that we're proud of

We tested the central claim instead of asserting it. The adversary in tests/test_anchors.py is twenty real sentences lifted out of a different paper in the corpus and attached to plausible values — correct register, correct vocabulary, wrong document, which is precisely what a hallucinating extractor emits. All twenty are rejected; anchor rate 0.0. The suite pushes from the other side too: line-wrapped quotes, ligatures and curly punctuation must still anchor, because a checker that rejects real evidence is broken rather than strict. 11 tests, no dependencies.

The demo page verifies its own cryptography. The hash chain is not a picture of a hash chain — it is recomputed with SHA-256 in the viewer's browser, and you can break it and watch it name the exact line that was edited.

What we learned

The valuable primitive was not better extraction — it was a cheap, deterministic check that the extractor cannot talk its way past. Everything defensible in this project follows from that one substring search.

What's next for Verbatim

Table-level anchoring using DWS structured-text bounding boxes, so a cell is anchored to coordinates rather than a sentence. Batch mode across a whole systematic review with a PRISMA-shaped summary. And scheduled SerpApi re-verification, because a review's evidence base starts going stale the day it is published.

Both integrations are live

Stated plainly, because a tool about provenance should have some.

Nutrient DWS. Every page of text in the demo came back from a real POST /build call carrying an ocr action and a json-content output. Each document's provenance line reads nutrient-dws, and one of the four is an image-only scan with no text layer at all, so DWS's OCR is what produced its words. The cross-check compares two renditions that were both read by DWS - and the live text surfaced real OCR artefacts our local fixtures had hidden: "committce" for "committee", "\ranian" for "Iranian", at 83-90% character similarity, with zero values lost.

SerpApi. Every retraction verdict came from a real search, logged with its exact query and timestamp. It discriminates correctly across the corpus: the two clean trials come back CLEAR, and the retracted Gulf War trial returns RETRACTION_SIGNAL, evidenced by PubMed's retraction notice and the publisher's own withdrawal. That study anchors 9/9 - every number in it is exactly where it claims to be - and it still must not enter anyone's meta-analysis. That gap between "correctly extracted" and "still trustworthy" is the entire reason the SerpApi layer exists.

The anchor checking, the cross-check, the evidence table and the hash chain all run for real as well; the chain is recomputed with SHA-256 in the viewer's own browser.

Built With

• javascript
• nutrient-dws
• ocr
• pymupdf
• python
• serpapi
• sha-256
• web-crypto-api