What inspired us

We kept running into the same fact: clinical trials get amended constantly, and every amendment costs real time and money. When we looked into why, a pattern showed up. A lot of the time the sponsor wrote the eligibility rules too strictly, barely anyone qualified, and a few months in they quietly loosened the rule. It’s something like “no prior chemo within 28 days” turning into 14, or a blanket “no brain metastases” getting softened.

Then we found out ClinicalTrials.gov keeps the full edit history of every trial. The evidence for which rules get loosened was just sitting there, public and unused. We figured we could teach a model to spot those rules before the trial starts.

What we built

Amendment Oracle takes a draft protocol and flags the eligibility rules that sponsors have tended to loosen later in similar trials. For every flagged rule you get a risk score and a handful of similar past trials, with their real NCT IDs. You also get a rough estimate of what an amendment could cost. There’s a “Review with agent” button too. It has Grok read the draft and write a plain-English review, and it checks its numbers by running code inside a Daytona sandbox.

How we built it

We started by pulling 1,500 oncology trials. Most are lung cancer (about 85%), and the rest are other cancers like leukemia and lymphoma. About 880 of them had more than one version of their eligibility text, and those were the ones we could learn from.

For each trial, we broke every version’s eligibility text into individual rules and matched them up across versions. That gave us about 24,600 rules, and for each one we asked whether it was loosened later. Then we trained a simple, explainable model on older trials and tested it on newer ones. That way it never gets to peek at the future. The cost estimate is basically the chance a rule gets loosened times what an amendment typically costs:

$$\text{expected avoidable cost} = P(\text{loosened}) \times \text{cost per amendment}$$

The agent side is deliberately locked down. Any code the AI writes runs in a Daytona sandbox with the network turned off. We only upload the data it needs and delete everything when it’s done. We also made the reviewer back up its claims, so every trial it mentions has to actually appear in the tool results. It can’t invent evidence.

What we learned

The biggest surprise was that most “changes” in the raw data aren’t real changes. Our first pass said hundreds of rules had been removed, when really they’d just been moved around or reworded. Cleaning up those labels did more for us than any fancy modeling.

We also learned to test on the future instead of a random split, because a random split looks amazing and tells you nothing. Once we did it properly, the results got humbler. On trials the model had never seen, its top-3 flags caught a loosened rule in about 44% of the trials that had one. Random picks would catch about 27%. That’s a real improvement but a modest one, so we call it a nudge for a clinician to take a second look, not a verdict. The dollar figure is an illustration built from published averages, not a forecast for any one trial.

The challenges

Getting the data was the hardest part. The version-history data loads fine in a normal browser but gets blocked for ordinary scripts. We ended up pulling it through a browser session in batches, and every time the page reset we had to pick up where we left off.

The text itself was a mess too. Some of it came as HTML and some as plain text, with random bullets, wrapped lines, and missing headers. Our first splitter turned an entire page into one giant “rule,” which was a fun bug.

Then there was the label cleanup, which took several rounds before the numbers stopped lying to us. On top of that we had to get the reviewer running on Grok, and we turned a wall of unreadable HTML error text into a message a person could act on.

What’s next

We’d like a clinician to hand-check a sample of our labels, expand beyond cancer trials, and look at numeric thresholds more closely. Right now we only know a rule changed, not exactly how much a number moved.