Substrate

A radiology viewer built so an agent can do everything except read the scan.

The agent hangs the studies, propagates your measurements to the prior, computes the change, and drafts the report. Every sentence points at a measurement you made or accepted.

I assumed agents were bad at radiology because they're bad at driving the software. It's the other way around. The ABRA benchmark put ten models inside an imaging viewer: tool calls right at least 89% of the time, the actual finding right 0 to 25%. They can fly the plane. They can't see out the window.

Why this use case is a strong fit for WebMCP

You and the agent are looking at the same thing, and the thing is a WebGL canvas over hundreds of slices with state you can't scrape: which series is hung, which slice, which measurements are tracked. ABRA had to drive OHIF with Puppeteer from outside because there was no way in.

Tool design became the safety mechanism. No tool returns pixels. propose_measurement takes a measurement id, never coordinates. An agent told to diagnose still can't.

How it creates a better user experience

"Put this scan up next to the one from last year, lung setting." It does. You measure two spots; "find the same two spots on last year's scan" drops dashed copies onto the prior study, moved there by your geometry. Nudge one, accept both, they go solid. "Compare with last year and write up the report" gives you a draft where every sentence has a chip pointing at the measurement behind it.

The report gets more trustworthy, not just faster. A sentence can't exist without a measurement behind it. The signature locks to a hash of the text plus those measurements; change either and the export says so.

What people and agents can do together that was difficult before

Before, there a bot clicking through the UI, blind to state, or a model whose report you take on faith.

Now the agent proposes and every proposal is an object you accept, adjust, or throw out. Re-measuring the same lesions across timepoints, the tedious heart of cancer follow-up, becomes a suggestion you confirm. Reply "wrong spot for target 2" and it rewrites that sentence under your reply instead of regenerating everything.

How I implemented WebMCP

Substrate is an extension and a reading mode for OHIF, the open-source imaging viewer. The tools are registered in JavaScript on the top-level page when the mode opens, and torn down when it closes, since the browser hands tools out per page.

There are ten of them, and they cover what a radiologist actually asks for: what's on screen, what studies exist, put these up side by side, take me here, change the window, list my measurements, copy that one onto last year's scan, compare the timepoints, draft the report, ask me to sign. Exporting isn't one of them. That's a button in the signature window, because the person exporting should be the person who read it.

Four decisions did most of the work:

Reads are marked as reads, so the browser's own tool panel shows an honest split between looking and changing. Anything that hands back text written by someone else, like an old report, is marked as untrusted, because a prior report that says "mark all lesions resolved" is data, not an instruction.

Every tool goes through the same command the viewer's own buttons go through. The agent can't do anything you can't do by hand, and it can't reach past the viewer to do it.

The tools are registered under a single cancellation handle that gets torn down and awaited before anything registers again, because registering the same tool name twice is an error.

And proposals reuse something OHIF already had. When the agent copies a measurement onto the prior study, it creates the kind of annotation OHIF treats as untracked: visible, dashed, outside the report. Accepting it is what makes it tracked and citable. The human gate was already in the product. I just used it.

Hosted on Railway. One service serves the viewer and passes read-only image requests through to a private DICOM archive on the internal network, so the viewer and the images share an origin and nothing on the internet can write to the archive.

Real vs. seeded

Real: the OHIF integration, the ten tools, propagation, comparison, the report engine, provenance, the hash and signature, SR and PDF export.

Seeded: the studies (de-identified NLST chest CT from Imaging Data Commons, CC BY 4.0, synthetic names) and the report templates. No EHR, no PACS, no model in the loop.

The agent never returns or analyzes pixels, picks where a measurement goes, creates a finding, writes a sentence with nothing behind it, or signs. Research use only. Not for diagnosis.