ScanProof

Your model says 99.9%. ScanProof says whether that's worth anything.

Comment

ScanProof — Project Story

Inspiration

The first thing I did was try to break a model on purpose.

I had a chest X-ray classifier fine-tuned and working: 94.2% accuracy, AUROC 0.992, the kind of number that goes on a slide. Then I fed it an image of breast ultrasound. Not a chest. Not even the right modality. Not the right organ.

It said pneumonia. 95.8% confident. That was higher than its average confidence on the chest X-rays it was actually trained for.

What unsettled me wasn't that the model was wrong. Models are wrong. It was that the model had no way of being uncertain about this. A two-class softmax normalises its probability across exactly the two classes it knows, so the two outputs always sum to 1 no matter what you put in front of it. There is no third slot, no output state that means "I have never seen anything like this." The probability mass has to go somewhere, so it goes to whichever class the features happen to lean toward, and if the input is far outside the training distribution that lean can be arbitrarily strong and arbitrarily meaningless.

Confidence isn't a lie the model tells. It's a question the architecture cannot be asked. Which means every calibration technique I knew about was solving the wrong problem: you can temperature-scale a softmax until its confidences are beautifully calibrated on the distribution you calibrated it on, and that does nothing for the input that arrives from somewhere else.

The failure this actually causes in the world is not dramatic. It's a model validated at one hospital, quietly run on another hospital's patients. Same modality, same anatomy, same projection. Different scanner, different protocol, different population. Nothing looks wrong, no error is thrown, accuracy degrades, and nobody finds out for months, because the thing that would tell you is a ground-truth label and that is exactly what you don't have in production.

So I stopped trying to build a better confidence score and started building the thing that sits in front of the model and refuses on evidence.

What it does

ScanProof takes a single chest X-ray and a model's prediction about it, and returns PASS, REVIEW or BLOCK with the measurement behind the decision. Four independent checks run on every prediction. Not one score with four inputs, but four checks that fail for genuinely different reasons:

  • Typicality asks whether the model has seen inputs like this. It measures the squared Mahalanobis distance from the image's feature vector to the nearest class-conditional Gaussian fitted on the training set, reported as a percentile so the number means something to a human without knowing the scale.
  • Stability asks whether the answer survives harmless edits. It re-runs the prediction across 7 label-preserving perturbation families at 3 severities each, 21 deterministic variants in all.
  • Agreement asks whether independently trained models concur. It compares 3 checkpoints that differ in architecture, seed and augmentation.
  • Confidence asks how far the prediction sits from the decision boundary.

Two hard gates override the weighted score entirely: an out-of-distribution gate at the 99.5th embedding percentile, and a fragility gate that fires when the label flips at the mildest perturbation severity. Some failures should not be averaged away by three other signals that happen to be content.

Every verdict comes with an evidence ledger where each deduction is a quoted measurement. Not "low reliability", but "prediction flips under Gamma/windowing gamma 1.90". I was strict about this. A guardrail that cannot tell you which check objected and what it measured is just a second opaque model stacked on the first one, and I would have built the exact thing I was annoyed about.

To show it works, I designed the most ordinary shift I could think of: train on children, deploy on adults. Fine-tune on pediatric chest films from one institution, then run on adult frontal chest films from a completely different one. Pediatric to adult:

  • Accuracy fell from 94.2% to 62.6%, a drop of 30.5 points.
  • Model confidence fell from 94.5% to 86.0%, a drop of 7.6 points.
  • The embedding percentile rose from 52.4 to 98.4, from the middle of the training distribution to its 98th percentile.
  • The ScanProof PASS rate collapsed from 61.1% to 4.8%.

Seven and a half points of confidence. On a live stream, with no labels, watching a dashboard, you would never notice that. It is inside the noise of a normal week. The model lost a third of its accuracy and its self-report barely flinched, while the guardrail withheld 95% of those inputs.

The detail that matters most: the observable numbers are all label-free. They are properties of the input and the model, not of the answer key, so you can compute them on the first patient of the first day. That is the entire point. Catch the problem before the accuracy report exists, because by the time the accuracy report exists it has already happened to people.

How I built it

PyTorch, FastAPI and React/TypeScript/Tailwind, served as one process with zero network calls on the demo path. Fonts are vendored, models are local, and datasets are fetched once with MD5 verification.

  • Data. PneumoniaMNIST, ChestMNIST and BreastMNIST from MedMNIST v2 (CC BY 4.0), pulled through the official medmnist API. No scraping, no private data.
  • Model. A 3-member ensemble, two ResNet-18 and one DenseNet-121, fine-tuned from ImageNet weights, with per-member temperature scaling fit on validation.
  • Evaluation. Thresholds were selected on the 524-image validation split and frozen before the 624-image test split or any shift arm was scored.
  • Reproducibility. make reproduce runs the whole path from a clean checkout, and every figure in the README and this submission is generated by make audit and make shift and committed under artifacts/.

The scoring module imports no torch and touches no disk, which is why the gate and arithmetic tests run in milliseconds and why I could test the decision logic with plain arrays rather than a GPU.

Challenges I ran into

A silent train/serve skew that ate a day. Validation accuracy sat at 74.6% and would not move. I retrained, changed learning rates, backbones and schedulers, swapped EfficientNet-B0 for DenseNet-121. Nothing. And the whole time AUROC was 0.88, so the model clearly knew something and just could not act on it.

The clue I walked past three times: 74.6% is exactly the majority-class rate. The model was not learning badly, it was predicting one class for everything and getting credit for the class imbalance. The cause was four lines apart in the transform pipeline. Augmentation was running after ImageNet normalisation, and torchvision's v2 photometric transforms clamp float tensors to the unit range. Normalised tensors have negative values, so every augmented training image was silently clipped back into unit range and un-normalised, while validation images went through untouched. Reordering two lines took validation accuracy from 74.6% to 98.5%.

The lesson was not about torchvision. I had spent a day debugging a silent distribution shift between training and validation, which is almost exactly the failure mode this project exists to catch. I got to experience my own problem statement from the inside, and after that ScanProof stopped feeling like a hackathon idea and started feeling like something I needed.

A confound that would have invalidated the whole study. The adult dataset is only distributed at 128 px unless you pull down 3.7 GB, so it gets resampled to 224. Which means my entire headline result had an obvious alternative explanation: maybe I just built a resampling detector. Nobody was going to catch that but me. I built a control arm anyway, putting the pediatric films through the identical 128-to-224 path. The PASS rate moves up by 0.96 points and accuracy down by 1.1, nothing next to a 30-point collapse. The effect is the patient population, not the pixels.

A build toolchain that failed silently, for contrast. @tailwindcss/vite emitted preflight and theme variables but zero utility classes under Vite 8, so the page rendered as unstyled HTML for an hour while I assumed I had broken React.

Accomplishments I'm proud of

Publishing the result that went against me.

Before running the shift study I wrote down what would count as success: a signal is acceptable in both regimes if it lands within 0.01 AURC of the best in-distribution signal and within 0.05 AUROC of the best shift-detection signal. I fixed those margins in advance specifically so that I could not move them later. Then I ran it. Lower AURC is better, higher AUROC is better.

Signal In-distribution AURC Shift-detection AUROC Worst case
Model confidence 0.0126 0.7479 0.203
Perturbation instability 0.0161 0.6941 0.000
ScanProof composite 0.0175 0.7955 0.383
Ensemble disagreement 0.0188 0.7113 0.065
Embedding percentile 0.0385 0.9592 0.000

Two results in that table are bad for me. On in-distribution data, plain confidence is a better error ranker than my composite score, 0.0126 against 0.0175. The baseline beat me. And no signal clears both margins, mine included. The hypothesis I pre-registered failed.

It was 2am and there was nobody to talk it through with, which is exactly what made it a real decision. Widening the margins by 0.01 would have made my headline claim true and absolutely nobody would ever have known. No teammate to object, no reviewer, no second pair of eyes on the commit. I went back and forth for maybe twenty minutes. What settled it was noticing what I was actually doing: sitting alone in a medical-imaging project, debating whether to quietly adjust a threshold after seeing the data. That is the precise behaviour that makes clinical AI untrustworthy, and building a tool that demands models show their evidence while hiding my own would have made the whole thing incoherent.

So the margins stayed, and the failure is printed on the audit page in the app under a heading that says "Reported as found." Not in a footnote. On the page a judge will open.

What the evidence does support is narrower, and I state it as such. Rescale each regime so the best signal scores 1 and the worst 0, then take each signal's lower score of the two. That is the question a deployed system actually faces, because it has to commit to one number without knowing which kind of failure walks in next. On that criterion the composite leads at 0.383 against 0.203, and the region that would beat it on both axes at once is empty. That is a claim about worst-case coverage, not dominance. ScanProof does not beat confidence in-distribution; it covers a failure mode confidence structurally cannot see, and blurring that distinction would have been the easiest lie in the project.

I am also proud that the credit is honestly assigned. On the adult arm the typicality sub-score fell from 0.878 to 0.116 while the other three moved by less than 0.19, so that one check did essentially all of the work, alone. That is stated in the UI rather than letting the composite take the credit. The other three stay because they catch cases this arm does not contain, and the demo deck ships exactly those: confident-but-fragile predictions where typicality is perfectly happy and the prediction still is not safe to use.

What I learned

No single signal is good at both failure modes. Confidence wins in-distribution and is blind to shift. Embedding distance is the exact reverse: AUROC 0.9592 at detecting the adult arm, and dead last at ranking in-distribution errors. They are not competing solutions to one problem. They are solutions to two different problems that happen to look alike from a distance.

Calibration is not the same as knowing what you do not know. I came in assuming a well-calibrated model would degrade gracefully off-distribution. It does not. Calibration is a property measured on a distribution, and carrying the model somewhere else does not carry the guarantee with it.

Every claim should be executable. Each figure in the README and this submission is generated from committed artifacts and verified by a test that parses the numbers back out of the prose and fails the build if they drift. Working solo, that test suite was the only reviewer I had, and it caught two real errors in my own writeup before I did. Documentation that cannot fail CI is documentation that is already wrong.

The honest number is more persuasive than the flattering one. Every version of this submission that hid the negative result was less convincing, not more, because a guardrail that hides its own author's failures is asking you to trust exactly the thing it claims to replace.

What's next for ScanProof

A learned, regime-aware weighting. The clearest remaining win is sitting in my own results table: averaging four signals dilutes the one carrying the shift. The raw embedding percentile detects the adult arm at AUROC 0.9592 while the composite manages 0.7955. A fixed weighted average is the wrong aggregator when the signals are good at disjoint things, so the fix is to infer the regime first and weight accordingly.

After that, prospective evaluation on real institutional data with a radiologist in the loop, and extending the check battery to modalities where "harmless edit" means something different than it does for radiographs. The perturbation families encode radiograph-specific assumptions about what leaves a finding intact, and those assumptions do not transfer for free.

Built With

Share this project:

Updates