Trialwright

• 

  1

• 

  2

• 

  3

• 

  4

• 

  5

• 

  6

• 

  7

• 

  8

• 

  9

• 

  10

• 

  11

• 

  12

Inspiration

Nearly half of the big rewrites to clinical trial protocols were avoidable. Tufts CSDD found that 57% of protocols get at least one substantial amendment and that 45% of those were judged avoidable. A single amendment costs a median of $141k in Phase II and $535k in Phase III, and the trial finishes about three months later. Every hospital has to re-approve it and re-train its staff.

Much of this starts with the document disagreeing with itself. A sample size that is 120 on one page and 100 on another. A dose that changed in the synopsis but not the schedule. Today, tired humans catch these by reading hundreds of pages. We wanted to build the colleague who reads all of it first.

What it does

Protocol Pre-Flight reads a trial protocol PDF and returns a list of issues. Each one says where it is (highlighted on the exact page), why it matters, and what to change. It also gives an overall pre-flight score.

• Completeness: checks the protocol against all of the ICH M11 template and the 34 SPIRIT 2025 items, marking each present, thin or missing.
• Consistency: cross-checks numbers, doses, ages and visit windows. It checks every endpoint against the schedule of assessments and recomputes the sample size.
• Burden and peers: adds up the patient hours from the schedule and compares them with similar trials on ClinicalTrials.gov. In our demo protocol, 25 exclusion criteria stands out against a median of 15.
• Co-author: from a short brief, it drafts M11 sections or USDM JSON, citing similar trials. The sample size comes from a calculator, never from the AI.
• Compare: run a revised protocol and see which findings were resolved, which are new and how the score moved.

It assists authors and does not approve protocols. The UI says so on every page.

How we built it

A Python/FastAPI backend and a Next.js frontend. PyMuPDF and pdfplumber read the PDF into a structured protocol graph (aligned with the CDISC USDM model) in which every fact keeps its page and position. Three things run on that graph:

1. A rules engine that does the actual error finding. It is deterministic.
2. An AI review committee of four reviewers (writer, scientist, statistician, operations) that adds advisory notes.
3. A verifier that drops any AI note whose quote isn't really on the page it cites.

The model never does arithmetic or scoring.

How we tested it

The starter sheet suggested planting errors in real protocols, so we built an injector that edits real ClinicalTrials.gov PDFs. It plants 16 kinds of error with committed answer keys. We tuned only on 3 development protocols and kept 7 back.

On those 7 unseen protocols with 65 planted errors:

On the protocols we tuned on, it found 70%, so 43% is the number to believe. The verifier dropped 60 of 446 AI notes, and none of the 386 shown cited text that wasn't on the page. It takes about 31 seconds and one cent per protocol. A clean clone with no API key reproduces these numbers, because the AI's answers are recorded.

We also looked back at 200 real melanoma registry histories. Trials that later relaxed their inclusion rules were about 4 times as likely to have been outliers before (p = 0.003, exploratory).

Challenges

• Real protocols are messy. We ran a real, amended TB protocol (TRUNCATE-TB, v1.0 and v2.0) that we hadn't tuned anything on. The first run was wrong in ways we could see. It read a heading "Arm F" as five arms. It flagged two treatment durations that the trial exists to compare. It counted authors' initials as abbreviations. We fixed those rule bugs, and the wrong findings dropped from about 23 to 2.
• Planting errors without leaking the answer. The injector has to redraw a line so that it looks untouched. An early version deleted neighbouring lines, which made precision look worse than it was.
• Keeping the AI honest. A bare AI prompt on the same protocols produced 1,159 findings, and 93 of them cited text that isn't on the page. That is why every AI note goes through the verifier.

What we learned

The rules found every error in our benchmark. The AI's value is in reading the protocol into structure and adding advisory notes, and only if it has to show its quotes. Measuring on protocols we hadn't tuned on cost us 27 points of recall, and it's the number we trust.

Limitations, said plainly

• 43% recall on planted errors in unseen protocols. Errors that need deep reading, such as a deleted endpoint, are mostly missed.
• Older, pre-M11 protocols produce false "section missing" findings.
• It doesn't compare the wording of two versions. Compare diffs the findings.
• Procedure times and costs are labelled assumptions.
• We haven't had clinicians check our flags, and we haven't compared against human reviewers.

What's next

Compare two protocol versions by their wording. Accept and reject buttons on the review screen, tied to the audit trail that already exists. Clinician-labelled flags and a human baseline.

Built With

• docker
• fastapi
• javascript
• llm
• next.js
• openrouteservice
• pdfplumber
• pydantic
• python
• railway
• sqlite
• tailwind
• typescript