Radar

Find sanctioned investigators, not their namesakes

Comment

Inspiration

Before a sponsor works with a clinical-trial site, someone has to check that the investigator has not been excluded by the HHS Office of Inspector General or debarred by the FDA. The lists are public — over 80,000 people on the OIG exclusion list alone — but they live on different sites, in different formats, and the check is usually done by hand, one name at a time, then rarely repeated.

What caught our attention in Regeneron's Red Flag Radar brief was the hard part hiding behind that chore: a name is not an identity. Site lists are full of typos, initials, swapped first and last names and nicknames; the sanction lists are full of namesakes. Miss a match and the trial is at risk. Flag a namesake and you wrong an innocent doctor. We wanted a tool that finds more true matches than a person would, accuses fewer namesakes, and explains every result so a human can decide.

What it does

  1. Upload a CSV of investigators (name, city, state, specialty, NPI if known) — or use the 300-investigator demo set.
  2. Match each one against 80,591 OIG LEIE individuals and 152 FDA debarment records.
  3. Score every candidate pair with a calibrated model: the probability that both records are the same person.
  4. Triage everyone into Probable match, To review or Cleared — the status is always written out, never shown by color alone.
  5. Explain each alert: both records side by side, the top reasons behind the score in plain English, the sanction type and date, and a link to the official record.
  6. Decide: a named reviewer confirms or dismisses, with a mandatory comment. Every decision is logged with the model and list versions used, and exported to CSV or PDF for audit.

The tool never decides. It points; a person decides.

How we built it

Data. We download the OIG LEIE CSV and parse the FDA debarment table, map both to one schema, and keep each record's source, URL, retrieval date and list version. Dated snapshots are committed so every result is reproducible; record ids are content hashes, so they survive list updates and keep the audit trail intact.

Normalization. Case, accents, titles (Dr, M.D.), suffixes (Jr, III), name order, and specialty vocabulary are normalized before anything is compared.

Blocking. Comparing 300 investigators to 80,000 records naively is 24 million pairs. We only compare records that share a Double Metaphone code of a surname part, or a short name-prefix key — which also catches swapped names and typos that change how a name sounds.

Features. Each pair gets 12 features: Jaro-Winkler similarity on last and first names, a nickname table (Bill = William), first and middle initials, swapped-name detection, phonetic agreement, NPI / state / city / specialty agreement — where missing stays missing, never counted as a match — and surname rarity from the 2010 US Census:

$$\text{rarity}(s) = -\log_{10} p(s)$$

where $p(s)$ is the share of the US population with surname $s$. A shared "Smith" ($\approx 2.1$) means far less than a shared rare name ($\geq 5$).

Model. A LightGBM classifier with monotone constraints — agreeing evidence can only raise the score — followed by isotonic calibration on held-out candidates, so a score means what it says:

$$P(\text{same person} \mid \hat{s} = 0.9) \approx 0.9$$

A logistic regression is kept as a reference. Explanations come from SHAP values on each pair, translated into sentences such as "Same NPI number — raises the score".

Thresholds. "Probable" starts at 0.90. The "To review" threshold maximizes the recall-weighted $F_2$ score on held-out training pairs — never on the test set:

$$F_2 = \frac{5 \cdot P \cdot R}{4P + R}$$

because missing a sanctioned investigator costs more than a second look.

Evaluation. The investigator side is simulated, as the brief asks. Our test set holds 30 real sanctioned people whose names we altered the way site lists mangle them, 40 invented innocent namesakes, and 230 others. Training pairs (~200,000) come from a disjoint pool of sanction records with a different random seed.

App. FastAPI + SQLite backend, React + Vite + Tailwind + Recharts frontend, Docker Compose with nginx on a Google Cloud VM, behind Cloudflare for DNS and HTTPS. One command runs everything from a clean clone.

Challenges we ran into

  • Being honest with our own test set. We fixed matching rules after looking at errors on the test set — so we generated three new test sets with fresh seeds and report those too.
  • Blocking misses. A typo in the first letters of a surname changed its phonetic code and the true record was never compared. Adding prefix keys cut missed positives in training from 4.5% to 0.2%.
  • Inverted names plus an initial ("MAZUR L.") slipped through swap detection and taught the model noisy examples; fixing it also removed a false alert.
  • Threshold plateaus. Isotonic calibration produces steps; our first threshold landed exactly on a plateau of common names in other states and flagged a dozen innocents. Choosing it by $F_2$ on training data fixed that without peeking at the test set.
  • Unstable ids. Row numbers change with every list update, which would break an audit trail — we switched to content hashes and found 33 strictly duplicated LEIE rows on the way.
  • Scarce identifiers. Only ~10% of OIG records carry an NPI and FDA records carry names only, so many decisions rest on name, place and specialty.

Accomplishments that we're proud of

  • It works on real data, end to end. 80,591 OIG and 152 FDA records, imported, normalized and searchable — with a live demo, not a mock-up.
  • 4× more sanctioned investigators found than an exact-name check (96.7% vs 23.3%), with zero namesakes accused — versus 27 for the exact-name check.
  • Numbers we can defend. A labeled test set the model never saw, a naive baseline, and three fresh test sets to show we did not tune to one draw.
  • Every alert explains itself in under 30 seconds: both records side by side, plain-English reasons, and a link to the official source.
  • A score that means what it says. Calibrated probabilities and monotone constraints: matching evidence can never lower a score.
  • Built for audit, not just for the demo. Stable record ids, dated list snapshots, and an append-only decision log with reviewer, time, model and list versions.
  • Fast and reproducible. 300 investigators screened in 1.7 seconds; the whole app runs from a clean clone with one docker compose up, backed by 18 automated tests and 17 test CSV files covering every upload path.
  • Honest about its limits. When two people truly share a name, middle initial, state and specialty, the app says so and asks for the NPI instead of guessing.

What we learned

  • Entity resolution is mostly about evidence you don't have: treating a missing NPI as "unknown" rather than "different" changed the model's behaviour more than any hyperparameter.
  • Calibration and monotonicity are what make a score explainable to a compliance officer, not just accurate.
  • A true namesake with the same middle initial, state and specialty cannot be told apart without an identifier — the right answer is to say so and ask for the NPI, not to guess.
  • For people-facing risk tools, wording is a feature: "To review", never "guilty".

What's next for Radar

  • Continuous re-screening: re-check active investigators on every list update and flag only what is new.
  • More sources: FDA clinical-investigator inspection outcomes, Form 483 observations and warning letters, weighted by severity.
  • Real identifiers: NPI registry lookups, single sign-on, and a pilot on a sponsor's real site list.

Investigator data in the demo is synthetic; sanction records are real public data. A match is a lead to verify, never a verdict.

Built With

Share this project:

Updates

Submission history