LeakCheck

LeakCheck catches clinical AI models that "predict" using data recorded after doctors already acted — exposing fake 0.97 AUCs as honest 0.60s.

Comment

Inspiration

A few months ago I went through a stretch of medical confusion of my own, and it pushed me to spend a lot of time trying to understand how hospitals actually use AI. That led to an earlier project: a sepsis prediction model, built from scratch, aimed at catching sepsis earlier than hospitals currently can.

While digging into why the leading deployed tool for this — used in 800+ hospitals, covering more than half of U.S. patients — performs so poorly in the real world, the answer wasn't really about model architecture. An independent academic study found the tool's honest, pre-treatment AUC was 0.63, far below the headline numbers usually associated with it. The reason: a lot of what the model was "learning" from wasn't available before a clinician already suspected the diagnosis. The model wasn't predicting sepsis. It was reading data that already gave the answer away.

That distinction — predicting versus reading-after-the-fact — felt bigger than one sepsis model. If this is happening, undetected, in one of the most widely deployed clinical AI tools in the country, the honest question is: what else is it happening to? Nobody is systematically checking for it. So I built a tool that does.

What it does

LeakCheck audits clinical AI models for one specific, well-defined failure mode: Clinical Evaluation Temporal Leakage (CETL) — when a model's reported accuracy depends on data that wasn't actually available before the moment it's supposed to predict.

Formally: a feature f is temporally contaminated if timestamp(f) ≥ timestamp(action) − δ, where action is the clinical decision point being predicted (e.g. a clinician suspecting sepsis and ordering treatment) and δ is a configurable contamination window.

Given a dataset of timestamped clinical features and a model's predictions, LeakCheck:

  1. Flags every contaminated feature, deterministically and transparently — no black-box judgment calls.
  2. Recomputes the model's AUC using only clean, pre-action data, and shows it side-by-side with the originally reported ("naive") AUC.
  3. Demonstrates this on a built-in synthetic case study modeled on the real Epic Sepsis Model finding, reproducing a comparable naive-vs-honest gap (0.974 → 0.599 in our reproduction) without requiring access to restricted clinical datasets.
  4. Lets users create an account, save scan history, and download a PDF report for any past audit.

How we built it

  • Detection engine (leakage_engine.py): pure, deterministic Python/pandas logic implementing the CETL definition — no ML involved in the audit itself, intentionally. An audit tool that itself requires trusting a black box defeats the point.
  • Case study generator (sepsis_case_study.py): synthetic timestamped patient data plus a real trained logistic regression (scikit-learn) — not a hand-tuned formula — that independently rediscovers the naive/honest AUC gap, used as the proof of concept.
  • API (api.py, FastAPI): exposes the engine over HTTP, including CSV upload for custom datasets.
  • Accounts and history: bcrypt password hashing, server-side session cookies, and a real SQLite database (database.py, auth.py) so users can save and revisit past audits.
  • PDF reports (pdf_report.py, reportlab): a downloadable summary of any saved scan.
  • Frontend (index.html): a single-file app — scan page with a live progress indicator, sign-in/registration, history, an About page, and an Inspiration page — built to read clearly rather than look like a generic dashboard.

Challenges we ran into

  • Making the synthetic case study honest, not rigged. Early versions of the synthetic data generator produced a naive model that scored a perfect AUC of 1.0 — which proved nothing, since real clinical signals are never that clean. We had to deliberately weaken and add noise to the "honest" baseline signal so the demo reflects a realistic gap rather than a number we engineered backward from the conclusion we wanted.
  • Getting the real-world citation right. It would have been easy to casually cite "the AUC went from 0.87 to 0.63" as if one paper proved exactly that mechanism. We checked the actual source (Wong et al., JAMA Internal Medicine, 2021) and made sure our paper and app represent precisely what it does and doesn't establish, rather than overstating it for effect.
  • Scoping the security work honestly. It was tempting to claim a fully "enterprise-encrypted" account system. Instead we built real, correctly-implemented fundamentals — bcrypt hashing, parameterized queries, HttpOnly session cookies, verified cross-user access control — and describe them accurately rather than oversell them.
  • Avoiding overclaiming the tool's scope. It would be easy to pitch this as a general AI-safety platform. We chose instead to keep it narrow and explicit about what it does and doesn't detect, because a tool that's honest about its own limits is more credible on a project whose whole premise is about catching dishonest evaluation claims.

Accomplishments that we're proud of

  • A precise, formal, nameable definition of a real failure mode (CETL) that didn't have a name before this project.
  • A working, end-to-end audit tool — not just a static script — with real accounts, persistent history, and exportable reports, built and tested in a single sprint.
  • A synthetic proof-of-concept that independently rediscovers a real, published finding, rather than a demo hand-built to match it.
  • Keeping every technical and statistical claim in the paper and app traceable to a verified source.

What we learned

That the most dangerous failures in deployed AI aren't always model failures — sometimes they're measurement failures, and those are much harder to notice because the model "looks" like it's working. Also, practically: synthetic data that's too clean teaches you nothing; the realism of the noise matters as much as the realism of the signal.

What's next for LeakCheck

  • External validation against a second, independent dataset (e.g. MIMIC-IV) to test CETL detection beyond the synthetic case study.
  • Extending the case study library beyond sepsis — readmission risk and 30-day mortality prediction are natural next domains, since both have the same "action point contaminates the evaluation" structure.
  • A standardized "leakage audit" report format that hospital procurement teams or EHR vendors could request alongside a vendor's standard performance claims, before adopting a clinical AI tool.

Built With

Share this project:

Updates

Submission history