SafetyNet

Half of hospital harm goes unnoticed. SafetyNet finds it before it's missed for good.

Comment

Inspiration

In July 2025, HHS's Office of Inspector General published a finding that stuck with us: hospitals miss about half of the harm events that happen to their own Medicare patients — not from malice, but because staff don't recognize the event as "harm" in the moment, or because it falls outside their hospital's narrow reporting criteria. About a quarter of Medicare inpatients experience harm during their stay, and 43% of that harm is preventable — meaning roughly 1 in 10 admissions involves something preventable that could easily go unflagged entirely. Hospitals already collect the data that would catch this (labs, meds, transfers, clinical notes) — nobody has the bandwidth to review every chart. We wanted to build a second pair of eyes for exactly that gap.

What it does

SafetyNet ranks hospital encounters by likelihood of undetected patient harm, so a reviewer with limited time can focus on the charts most worth a second look instead of reviewing randomly or chronologically. It works in three layers: a structured data watcher scanning labs, medications, ICU transfers, and readmissions for known red-flag patterns; a clinical note reader that scans free-text notes for harm-adjacent language, distinguishing "known complication" phrasing from "this wasn't supposed to happen" phrasing; and an interpretable scoring layer (logistic regression / decision tree) that combines both into a confidence score — always paired with the specific evidence behind it, never a black-box percentage. It's explicitly framed as triage, not diagnosis — the same category of tool as existing sepsis or fall-risk alerts.

How we built it

Real, labeled harm events are rare. To train anything we needed volume and enough positive cases, so we built a synthetic warehouse named SafetyHops, Synthea-style in SQL Server. Each encounter has labs, medications, procedures, and a harm tag from planted conditions (things like post-op bleed or opioid oversedation). Hospital CSV extracts are a different path: unlabeled stays that we rank after discharge, not use as training data.

The engine is currently two layers. A structured watcher looks for red flags a trigger tool would catch: reversal drugs, unplanned escalation, abnormal labs. Then a scikit-learn model (logistic regression, or a shallow tree) turns those signals into a ranked list and shows why a stay is high: not “87% risk,” but “unexpected ICU transfer and an abnormal post-op lab.”

Python runs the models. FastAPI is the shared service. Streamlit is the diagnostics dashboard (metrics, coverage, the note reader). React/Vite is the review queue. Both frontends call the same API so we didn’t duplicate scoring logic.

Structured data lives in SQL Server. SafetyHops is one database; hospital stays go into a canonical core schema.

Challenges we ran into

The biggest one: our model was scoring a suspiciously good 0.946 ROC AUC. We built a leakage auditor and found our own synthetic data generator had tied every harm label one-to-one with its own evidence: the model was reading the answer key, not learning anything. We fixed the generator to inject silent harm (real harm with no obvious trace) and benign triggers (a flagged medication given for a legitimate reason), and reported the honest, lower score of about 0.80. Separately, our synthetic cohort had a 46% harm rate, which quietly made the whole problem too easy — we resampled down to the ~10% preventable-harm rate the OIG report itself implies, dropping cases only (never duplicating, to avoid reintroducing leakage across train/test). On the infrastructure side, two teammates running different local SQL Server instance names kept breaking each other's hardcoded connection strings, which we only fully resolved with a proper .env-based config layer partway through.

Accomplishments that we're proud of

Catching our own data leakage before a judge could, and reporting the corrected, worse-looking number instead of the flashier one. Resampling to a realistic base rate and getting a defensible, honestly-measured lift over random chart review at that rate. Being willing to show the less flattering number, with the reasoning behind it, is the thing we're most proud of.

What we learned

That a suspiciously good metric is a bug report, not a win. The instinct to double-check an unusually strong number instead of screenshotting it for the pitch deck. That base rate quietly changes what every other metric means, so a lift number is meaningless without saying what it's a lift over. And, less glamorously, that per-developer environment configuration (differing local SQL instance names, missing drivers) can eat as much hackathon time as the actual modeling work if you don't solve it early.

What's next for SafetyNet

Getting real, credentialed access to full MIMIC-IV/eICU (rather than the open demo subsets) and to labeled clinical note corpora like n2c2's adverse-drug-event annotations, to validate the note reader against real, non-synthetic ground truth. We also want to expand our triage to include an NLP classifier that would be able to scan lab-text for concerning language that may indicate missed harm. And connecting the scoring output to an actual reviewer workflow (assignment, disposition tracking, outcome feedback) so the tool closes the loop rather than just producing a ranked list.

Built With

Share this project:

Updates

Submission history