Glass-Box CT

A trust layer for oncology imaging AI: it flags when a CT model’s tumour measurement can’t be trusted, explains why with Nemotron agents, and shows the interpretability evidence behind it.

Comment

Inspiration

Cancer treatment response is judged by RECIST: measure the longest diameter of each target tumour, add them up, and compare scans over time. AI segmentation models can now do this in seconds, and they report a single confident number.

We wanted to know when that number is wrong. We tested NVIDIA's open CT foundation model, NV-Segment-CT, on 75 held-out liver cancer patients it was never trained on. It under-measured the sum of tumour diameters by 19% [95% CI −28%, −13%], never over-measured, and essentially missed about 1 tumour in 5. Nothing in its output warned you.

$$ \text{bias} = \frac{\sum d_{\text{model}} - \sum d_{\text{reference}}}{\sum d_{\text{reference}}} $$

An error that size can change whether a treatment looks like it's working. So we built a layer that tells you when not to trust the measurement, and we opened the model up to find out why it fails.

What it does

Glass-Box CT is a trust layer for oncology imaging AI. For each CT scan:

  • Segments and measures: NV-Segment-CT outlines the liver and tumours, and we compute RECIST target lesions and the sum of diameters.
  • Gives a trust verdict: a quality flag reads the model's own internal signals (confidence and uncertainty inside its predicted tumour) and returns Trusted / Review / Unreliable. It catches segmentation failures with AUROC 0.93 [0.86, 0.98].
  • Suggests a correction, honestly labelled: when the flag fires, a flag-gated recalibration shows what a lower threshold would report, next to the original and never replacing it, with its over-estimation cost stated on screen.
  • Explains it with agents: NVIDIA Nemotron models on Nebius Token Factory write the report and answer questions. A fast Nemotron (Lightning) calls our measurement tools; Nemotron Super writes the text; every number is checked against the tool outputs before it's shown.
  • Records everything: an audit record (model and checkpoint hash, thresholds, flag reasons, the agent's report and tool-call log) exports as PDF or JSON with a checksum.
  • Shows the evidence: an Evidence page lays out the research behind every rule, with confidence intervals and links to the code.

How we built it

  • Model: NVIDIA NV-Segment-CT (VISTA-3D, 132 classes), with hooks into its encoder and decoder features.
  • Interpretability: linear probes with random-init and raw-intensity controls, LEACE concept erasure, activation patching, logit attribution and brightness counterfactuals, all pre-registered before running, with every deviation logged.
  • Agents: Nemotron Lightning (tool calling) and Nemotron Super (writing), served through Nebius Token Factory, which is the only LLM provider in production.
  • App: FastAPI backend; React, Vite, Tailwind and Framer Motion front end; NiiVue for CT viewing; Docker on Render, with demo data pulled from a checksummed GitHub release.
  • Data: public, de-identified datasets only (HCC-TACE-Seg held out; training-data audit on MSD Liver and CRLM-CT).

Challenges we ran into

  • Data is messier than it looks. Our DICOM converter mixed up multi-acquisition series, and one label file declared a 25 mm offset that common readers ignore. We built alignment checks that caught both, fixed them at the source, and re-ran everything.
  • Honest numbers over flattering ones. Our flag-gated recalibration looked great in-sample; under nested cross-validation it was good but not clearly better than always recalibrating. The app shows the cautious numbers.
  • Some ideas didn't survive. A contrast-phase heuristic agreed with the reference at chance level (κ = 0.04), so we removed it. Steering the model's features turned out to be equivalent to a threshold shift, which led us to the recalibration design.
  • Keeping LLM output trustworthy. Agent reports are only shown when every number matches the tool outputs; otherwise the app falls back to a labelled template.
  • Shipping 3D medical data on a small server. We moved the demo data into a checksummed release that the Docker build downloads and verifies.

Accomplishments that we're proud of

  • A finding, not just a demo: a held-out, pre-registered result showing where a public CT foundation model fails, and by how much.
  • We explained the failure, not just flagged it: probes, erasure and counterfactuals trace the misses to a tumour concept the model holds but doesn't use for bright tumours.
  • It held up beyond one model: in follow-up research the same blind spot appears in all three CT models we tested, and retraining with synthetic bright tumours raised bright-tumour detection from 21% to 86–89% across two independent training runs.
  • Honesty built into the product: every verdict comes with its reasons, every correction with its cost, and every agent number is checked against a tool.
  • Shipped end to end: a live app on Nebius Token Factory with NVIDIA Nemotron, open source under MIT, with a full test suite and a public audit trail of every experiment.

What we learned

  • The failure isn't random. Tumours brighter than the surrounding liver are the ones the model misses.
  • The model sees them: a probe on its internal features detects the missed tumour tissue (0.60–0.63, against about 0.08 for the controls). The information is there; the final readout doesn't call it tumour.
  • Brightness causes it: editing a dark tumour to be bright made the model lose it in 56 of 62 cases. Erasing the tumour concept cut tumour Dice by 0.33 while leaving the liver untouched.
  • The training data explains a lot: in the public liver-tumour data the model drew on, only about 1 in 20 tumours is brighter than the liver (5.2%). In our held-out cohort it's about 1 in 3 (32.6%).
  • Confidence scores aren't enough. A model's own internal signals, read the right way, are a much better guide to when to trust it than its output alone.
  • Pre-registration pays off. Writing the rules down before running caught our own mistakes and kept the reported numbers honest.

What's next for Glass-Box CT

  • Publishing the full research paper in November 2026.
  • Validating on real multiphase CT: arterial vs portal-venous scans of the same patients, not just synthetic edits.
  • Packaging the audit as a tool any team can run on its own segmentation model to find contrast-dependent failures.
  • More organs and tumour types: our kidney experiments show the same brightness vulnerability with a different internal mechanism, so each needs its own check.

Research use only. Not a medical device and not for clinical decisions.

Built With

Share this project:

Updates

Submission history