SiteLine

Inspiration

Before a clinical trial can enroll a single patient at a hospital, the sponsor and that hospital have to agree on a clinical trial agreement. A sponsor sends the same form contract to dozens of sites, and each one sends it back with its own redline. Contract staff then review every tracked change by hand, including the many that the sponsor's own negotiation playbook already answers. Across 61 cancer centers, study start-up took a median of 167 days against a 90-day target, and slow site activation is linked to missed enrollment goals (Contemp Clin Trials Commun, 2025).

We wanted to understand what that delay actually costs, so we priced it. Each week a site's contract stays open pushes the whole trial back by that site's share of enrollment, and a day of trial time is worth the chance the drug is approved times its daily sales (discounted back to today), plus the direct cost of running the trial:

$$ \text{Cost of delay per week} = 7 \cdot w \cdot \left( \frac{p \cdot S}{(1+r)^{T}} + C \right) $$

where $w$ is the site's share of the study's enrollment rate, $p = 9.3\%$ is the Phase II solid-tumor chance of approval, $S = \$840{,}000$ is median oncology sales per day, $r = 10.5\%$ is the discount rate, $T = 7.6$ years to launch, and $C = \$23{,}737$ is Phase II trial cost per day (Tufts CSDD 2024; BIO/Informa/QLS 2021; DiMasi 2016). For our demo site, that comes to roughly $79,000 for every week its contract sits in someone's queue. Most of that queue is work a well-written playbook has already decided, and that is the problem Siteline goes after.

What it does

Siteline turns a sponsor's past negotiating positions into a central playbook and applies it to every redline that comes in. A contract manager uploads a site's Word redline, or the site simply emails it in, and within seconds every tracked change is sorted into one of four outcomes: Accept, Fallback (send the playbook's counter-language), Escalate to an approver, or Human review.

Each decision shows the rule that fired, the values pulled from the clause, and a verbatim quote from the contract that Siteline has checked against the document. The review screen is the contract itself, shown in full, with every redline tinted by its outcome so a change is always judged alongside the clauses around it.

Escalations go to the right person's phone. Legal, finance and privacy approvers each get an iMessage and reply 1 (approve), 2 (counter) or 3 (reject), and the decision updates live on the review screen. If they don't answer, Siteline calls them. When every change is settled, Siteline exports the counter-proposal as a Word file with real tracked changes and comments, ready to send back to the site.

Across the portfolio, a ranked view shows which sites are blocking activation, how many clauses are stuck, and how much money each one is costing per week.

How we built it

The core design rule is that the AI never makes the decision. It reads the contract and reports facts, and deterministic code decides what happens next.

Parsing in isolation (Daytona): each uploaded .docx is parsed inside a fresh Daytona sandbox with networking turned off. Only JSON comes back out, and the sandbox is deleted afterwards. The app shows each step with its measured time, so a sponsor can see the contract never touched the open internet.
Extraction: an LLM (GPT-5, with a Claude Code fallback through claude -p) reads each change and fills a strict JSON schema with clause values. Every value has to come with a verbatim quote, and we check that quote against the document text.
Rules engine: a 9-clause, 37-position playbook that we wrote the way a sponsor's contracts team would, with standards, counters, escalation triggers and talking points. The numbers in the playbook are exactly what the engine reads, and every decision carries a rule ID. Anything uncertain (a quote that fails verification, confidence below 0.80, a term no rule covers, an error) goes to a person and is never accepted automatically.
Approvals (Photon 2.0 + voice): one iMessage per escalation, with a phone-call fallback.
Precedents (Supabase Postgres + S3): terms from agreements that people have confirmed become precedents for future negotiations. Siteline never learns from its own automated decisions.
Frontend: React 19 + Vite. Backend: FastAPI on Python 3.12.
Evaluation: we built a labeled set of 243 redline changes and compared Siteline against an LLM-only baseline that receives the same playbook and simply decides on its own. The same comparison is wired into Chalk Evaluations.

Challenges we ran into

Our first full eval scored 79.0%, and most of the failures came from small, very literal problems. Twenty-two rows failed only because the model wrapped verbatim quotes in curly quotation marks. One clause had its carve-out narrowed to "intentional fraud", the model rounded that to "gross negligence", and it was accepted, which was our only unsafe accept. We fixed that by requiring the quote for a negligence carve-out to actually contain those words. Changing only a notice period was being read as granting termination for convenience, which caused eight false escalations. Each fix made the system stricter about what counts as evidence, not more clever.

We also ran into rate limits, because the full extraction prompt with the playbook is about 9,400 tokens. To keep iteration fast, we cache each model answer under a hash of its exact prompt. That brought a full re-run of the eval from over a minute down to about two seconds, while the quote checks and rules still run fresh every time. Finally, Photon 2.0 works very differently from classic Photon, so we built the approval flow directly against its new API.

Accomplishments that we're proud of

After those fixes, Siteline reached 94.2% accuracy on the 243-change set, against 91.8% for the LLM-only baseline. The numbers we care about most are about safety:

	Siteline	LLM-only baseline
Unsafe accepts (should escalate, was accepted or countered)	0	2
Routine changes escalated unnecessarily	0	7
Escalations that were correct	100% (73/73)	88.1% (74/84)
Escalation recall	94.8%	96.1%

The baseline catches slightly more escalations, but it also lets two through without an approver and wastes approvers' time on seven routine changes. Every case Siteline didn't escalate went to human review rather than being decided, so nothing was settled without a person or a rule standing behind it. In contract work that is the trade we want to make.

We're also proud that every piece of the demo is real end to end: real tracked changes in a full 17-article fictional agreement, real texts to real phones, and a real Word file at the end.

What we learned

The most useful thing we learned is that, in a high-stakes domain, the model's job should be as narrow as possible. Once the LLM only had to find and quote facts, and plain code decided what to do with them, we could test every rule, explain every outcome, and point to exactly where a mistake came from. We also learned that an honest baseline matters. An LLM given the playbook is already fairly accurate, so the real case for Siteline is that it is safer and easier to audit, not that it is smarter.

What's next for SiteLine

We want to support PDF and scanned redlines, extend the playbook beyond nine clause types, and let precedent from confirmed agreements suggest better counter-language for each specific site. We'd also like to pilot it with a real sponsor's contracts team on historical agreements, since that is the only way to measure how many days it actually takes off study start-up.

All contracts, sites and sponsors in this project are fictional.