Clinova

Clinical Trial Simulation Engine. Predict where clinical trial protocols will stall enrollment or lose patients, before launch, and validate every prediction against real ClinicalTrials.gov outcomes.

Comment

Inspiration

Many clinical trials fail before they test their drug. They fail because they can't enroll patients fast enough, or because the patients they do enroll drop out. In non-small cell lung cancer (NSCLC), the pressure is higher. Patients are sick, biomarker requirements shrink the eligible pool, and every month of delay costs time that patients don't have.

Many of these failures are visible in the protocol on day one. A narrow ECOG window, a long washout period, a demanding visit schedule, or a stack of exclusion criteria can each shrink the eligible pool or push people out. Sponsors usually learn this the hard way, through amendments, extended timelines, or terminated trials. We wanted a tool that gives the warning before the first patient is screened.

What it does

Clinova takes a draft NSCLC trial protocol and predicts where it will stall enrollment or lose patients.

  • Parses the protocol into structured eligibility criteria and a visit schedule.
  • Estimates a cohort funnel. It shows how each criterion narrows the eligible population.
  • Simulates patient behavior with agents that screen, enroll and drop out. This surfaces the burden points a static score would miss.
  • Predicts risk with a model trained on historical ClinicalTrials.gov trials.
  • Proposes edits with an optimizer that suggests protocol changes, such as loosening a criterion or consolidating visits. It re-scores each change.
  • Proves itself. Predictions are evaluated against real ClinicalTrials.gov outcomes on a held-out, time-separated test set.

Clinova advises. It never approves a protocol, and it uses no real patient data.

How we built it

We split the work into modules, each with a clear owner and a shared contract:

  • Ingest pulls ClinicalTrials.gov records into a database, including each trial's version history.
  • Parse uses an LLM to turn free-text criteria into structured fields.
  • Cohort and simulation estimate the eligible pool. Agent-based simulation, with a fast surrogate model, explores enrollment and attrition dynamics.
  • Model is the risk predictor. Conceptually it estimates

$$ P(\text{stall} \mid x) = \sigma!\left(w^\top \phi(x_{\text{v1}}) + b\right) $$

where φ(x⁽¹⁾) are features taken only from the trial's first registered version.

  • Optimize searches over protocol edits to lower predicted risk, subject to constraints.
  • API and app expose everything through a web interface.

Some engineering rules shaped the whole build:

  • Every LLM call goes through one wrapper that caches responses and enforces a call budget. This also makes the zero-network demo mode work.
  • Tests never touch the network.
  • A single script checks that every number in our docs was produced by code.

Challenges we ran into

  • Leakage. ClinicalTrials.gov records get edited after the fact, and later versions can encode the outcome. We restricted features to the first registered version and split by time, so training uses trials that started before 2018 and testing uses trials that started on or after 2018.
  • Honest evaluation. It's tempting to tune until the test score looks good. We committed to never tuning on the test split and to reporting numbers as they come out.
  • Messy eligibility text. Criteria are written inconsistently. Parsing them reliably with an LLM, within a cost budget, took careful caching and validation.
  • Simulation versus prediction. Agent simulations are expressive but slow and hard to validate. A surrogate model bridged that gap, but it needed careful checking against the full simulator.
  • Keeping docs truthful. Our number-checking script initially flagged mismatches between prose and results. Getting it to zero meant fixing the docs, not the numbers.
  • Licensing. The simulation inspiration, MiroFish, is AGPL-3.0, and this repo is MIT. We wrote our own client and copied none of its code.

Accomplishments that we're proud of

  • An end-to-end pipeline, from raw protocol text to a risk estimate and suggested edits.
  • A validation setup that avoids leakage by design: first-version features, a strict time split, and no test-set tuning.
  • A reproducible, network-free demo that runs from a seeded database.
  • A docs-versus-results check that blocks any hand-written number from shipping.
  • A multi-agent build with clear ownership, contracts and merge gates that held together.

What we learned

  • In applied ML on registry data, how you split the data matters more than which model you pick. A careless split can make a model look much better than it is.
  • Hard constraints, such as one LLM gateway, one budget and one source of truth for numbers, let many contributors work in parallel without drift.
  • Simulation helps most as a way to explain why a protocol is risky, not only as a score.
  • Being upfront about limits builds more trust than a headline metric does.

What's next for Clinova

  • Beyond NSCLC. Extend to other oncology indications and then to other therapeutic areas.
  • Richer inputs. Add site-level and geographic feasibility, real-world prevalence data and biomarker testing rates.
  • Better causal estimates. Move from "this protocol looks risky" toward "this specific edit reduces risk by about this much", with proper uncertainty.
  • Tighter loop with experts. Put clinical operations and biostatistics reviewers in the loop to check parsed criteria and suggested edits.
  • Prospective validation. Score protocols before they open, then follow their real enrollment outcomes.

Built With

Share this project:

Updates

Submission history