Second Opinion

A diagnostic model that reports when it should not be trusted — it withholds the diagnosis instead of guessing.

Comment

📄 One-page project description (PDF) · 💻 GitHub repository

Inspiration

Say you build a model on breast tumour cells, and it works. Then someone feeds it a lung tumour cell. Both are tumour cells. Under a microscope they look similar. But they are not the same thing, and a model trained on one has no business answering about the other.

Almost every health-AI demo will answer anyway — same confident percentage, no warning. That is not a rare edge case. It is the normal way these models fail when they leave the hospital they were built in: a model validated at one site degrades at the next, and nothing in its output says so.

I wanted to build the opposite: a model whose headline output can be silence.

What it does

Second Opinion answers three separate questions about every sample, and withholds the diagnosis when the last two say it should.

  1. What is the probability? — isotonic-calibrated logistic regression. Always answers, even for nonsense.
  2. Is that probability trustworthy for this sample? — split conformal prediction, which uses held-out data to derive a threshold and returns a prediction set with a coverage guarantee.
  3. Is this sample even the kind of thing I have seen? — class-conditional Mahalanobis novelty detection, which asks whether the combination of measurements is strange, not just whether any one value is.

None of them is enough alone. Reporting all three, and letting the last two veto the first, is the project. So the model has four possible outputs, not two: {malignant}, {benign}, {malignant, benign} (genuinely borderline — a human should look), and {} — abstain.

That empty set is the point.

When it withholds, it does not stop at "I don't know." The two kinds of withholding get two different, computed next steps:

  • a conformal abstain names the smallest single measurement change that would make the call decisive — "if mean concave points rises 32%, this becomes a 90%-confident malignant";
  • a novelty withhold names the measurement that looks like a typo, and refuses to offer a diagnosis on the far side of that edit.

The sharpest demo is the Implausible geometry preset: mean area at the dataset maximum while radius and perimeter sit at their medians. Every value is individually in range — a per-feature bounds check passes it without comment — but the combination is geometrically impossible. A nucleus cannot have that area at that radius. The classifier underneath is 99.61% confident it is malignant. The system reports no diagnosis, because Mahalanobis distance accounts for how features covary and puts the sample 52× beyond the radius containing 99% of training data.

That 99.61% is exactly what a conventional demo would have shown you, with nothing attached to tell you not to believe it.

How we built it

Python, scikit-learn, Streamlit. Wisconsin Diagnostic Breast Cancer: 569 samples, 30 features, 37.3% malignant. Split three ways — train / calibration / test — where the calibration split is never trained on and the test split is touched only for reported numbers.

  • conformal.py — split conformal prediction (LAC), finite-sample corrected
  • ood.py — class-conditional Mahalanobis novelty score
  • counterfactual.py — the computed next step for each kind of withhold
  • diagnostics.py — chart data, metric guides, honest probability formatting
  • shift.py — noise and lab-scale stress tests
  • scripts/evaluate.py — regenerates every number claimed, over 40 random splits
  • tests/ — 32 claim-level tests, one per claim made in the README or the UI

Every method traces to a published source: Sadinle, Lei & Wasserman (2019); Angelopoulos & Bates (2023); Lee et al. (2018); Guo et al. (2017); Street, Wolberg & Mangasarian (1993).

Challenges we ran into

Calibration and accuracy are different quantities, and I had to pick. Isotonic calibration is a monotone reshape of the probability. It cost a little AUC (0.994 → 0.985) and bought a better ECE (0.032 → 0.026). For a system whose entire premise is that the probability means what it says, that was the right trade — but it is a trade, so I report both columns.

A more flexible model made things worse. I tried a per-feature spline GAM expecting it to win. A controlled 25-seed A/B over an identical pipeline said otherwise: linear won on all eight metrics (ECE 0.031 vs 0.111, Brier 0.022 vs 0.049, coverage 93.5% vs 91.4%). At n=569 the extra flexibility is not supported by the data, and it is calibration that pays — the one property this project rests on. I reverted it and kept gam.py as the record of what was tried.

One split is a coin flip, not a finding. Conformal coverage on a single split of 569 samples ranges from 86% to 99% depending only on the random seed. Quoting any one of those would be cherry-picking, so every headline number is averaged over 40 random splits and reported with a standard deviation.

A bug that only showed up on real samples. The UI rebuilds 30 features from 10 sliders. It anchored each sample's error/worst columns on the dataset median, which double-counted a large nucleus's size and pushed it off the training manifold — so selecting "Typical malignant nucleus" falsely reported out-of-distribution at 4.2×. The existing test only covered a median start, the one case where both formulas agree. Fixed, with a regression test asserting every preset assesses identically to the raw sample it came from.

Accomplishments that we're proud of

  • The empty prediction set works end to end: 100% of deliberately scrambled inputs are flagged, versus 6.8% of genuine held-out samples.
  • Under progressive corruption the withhold rate climbs to 100% while confident error — decisive, in-distribution, and wrong — does not climb with it. It fails by going quiet rather than by guessing.
  • Conformal coverage 93.5% ± 3.5% against a 90% target, decisive on 95.1% of cases.
  • A reported probability is never 0% or 100%. Isotonic calibration returns a pure bin's observed frequency, so it can output exactly 1.0 — printed as "100%" that claims certainty a 128-sample calibration split cannot buy, so the reported value is clipped to 1/(2n) off each end.
  • Every claim in the README and the UI has a test asserting it, and every number regenerates with one command.
  • I reverted a model I had already built because the measurement said to.

What we learned

That the hard part of trustworthy medical AI is not making the model more accurate — this one is already at 0.985 AUC — but making it stop talking on the cases where accuracy is not the operative question. Abstention has to be a first-class output, designed into the interface, not a confidence threshold bolted on afterwards.

What's next for Second Opinion

  • Subgroup-conditional coverage. The conformal guarantee here is marginal (Angelopoulos & Bates, 2023): it holds across the population, not necessarily within every subgroup. Making it conditional is the obvious next step and is not implemented.
  • A second dataset, to test whether any of this transfers at all.
  • Upstream from the measurements — the real clinical problem is getting from a slide to the 30 numbers, which this prototype assumes already solved.
  • Clinician-in-the-loop evaluation: does a computed next step actually change what a human does with a withheld case?

⚠️ Research prototype, not a medical device. Trained on 569 historical fine-needle-aspirate measurements from one institution, collected in the early 1990s. It cannot diagnose anyone, has never been clinically validated, and must not be used for a real medical decision.

Built With

Share this project:

Updates

Submission history