Elsewhere: is this device failing elsewhere?

Inspiration

When an infusion pump or ventilator misbehaves, a hospital's biomedical engineer has to decide fast: is this one bad unit, or a failure happening in other hospitals too? The FDA publishes millions of device failure reports in its MAUDE database, but the useful detail sits in free-text narratives, and the structured problem codes that existing tools count miss most of it. Nobody on a hospital safety team has time to read thousands of reports per incident.

What it does

A nurse writes one line about a device that misbehaved and picks its asset tag. Elsewhere:

1. structures the report with Nemotron 3.5 Lightning,
2. matches it to a failure code with Nemotron 3 Super,
3. counts matching FDA reports for that manufacturer and nationally, week by week, and tests whether they are rising,
4. triages with Nemotron 3 Nano,
5. only when reports are rising, searches dated manufacturer notices with Tavily, puts matching FDA recalls first, and has Nemotron 3 Ultra rule on the cluster (emerging signal, known issue or reporting artifact), with every quote checked against the report it cites,
6. drafts a MedWatch report under 21 CFR 803.30, a pull list of the hospital's units of that model, and a work order. A human approves; nothing is submitted.

A full live check costs under 5 cents. When reports are flat, the Ultra ruling never runs and the check costs about a tenth of a cent.

How I built it

I pulled 229,000 MAUDE reports for infusion pumps, patient monitors and ventilators from openFDA, dropped reports filed because of a recall so they could not leak the answer, removed duplicates and masked brand names, leaving about 115,000. Lightning read every narrative with reasoning switched off, and a grounding audit checked each extracted failure mode against the source text. Super wrote failure codebooks per device type. Detection tests weekly growth in count and in share of a manufacturer's reports, so a surge in total reporting does not look like a new failure.

I pre-registered the backtest before running it: 46 held-out recalls, 3 control manufacturers, fixed thresholds, and baselines including PRR, the standard disproportionality method. The live API is FastAPI streaming each step over server-sent events, with per-run, daily and total spend caps and a cache, so a repeated incident costs nothing. The site is Astro with Preact for the incident console, and the charts are SVG rendered at build time. It all runs as one Render web service.

Token Factory features I relied on: structured JSON output, the per-call reasoning toggle, and prefix caching (a 227,000-token prompt went from 12.5 s to 1.1 s on repeat). The whole project cost about $23 of Token Factory usage.

Results

• Elsewhere caught 12 of 46 held-out recalls with 103 alerts; PRR caught 15 with 798. 14% of Elsewhere's alerts matched a recall, against 3% for PRR. PRR fired earlier, with a median lead of 34 weeks against Elsewhere's 8.5.
• FDA problem codes undercount the failures in the text. After hand-checking 50-report samples from each side, about 7,200 reports described the recalled failure according to Elsewhere's codes, against about 2,200 by FDA problem code, roughly 3 times as many.
• On the 13 recalls after the models' training cutoff, the pre-registered method caught none. A post-hoc check traced most of that to a codebook frozen in mid-2024: refreshed codebooks caught 4 of 13, against PRR's 7. Both results are published.

Challenges I ran into

My first extraction prompt included examples, and the model copied them into reports that did not describe them. A grounding audit caught it; I rewrote the prompt and re-ran the corpus. In the first experiment, raw report counts looked like perfect early warning, until I saw the manufacturers' total reporting had surged 10 to 25 times. I switched to share-based detection and narrowed the headline claim. With reasoning on, Ultra never finished its structured ruling within 16,000 output tokens, so live rulings run with reasoning off. Batch job creation on Token Factory was unavailable for my account, so the corpus ran as concurrent real-time calls.

What I learned

Most of the signal in device safety data is in the narrative, not the codes. A model that reads it needs auditing as much as prompting: two of my biggest findings were errors in my own first attempts.

What's next

More device types, a second rater for the coverage audits, codebooks refreshed on a schedule, and the batch API for the weekly corpus update.

Built With

• astro
• fastapi
• nebius-token-factory
• nemotron-3-nano
• nemotron-3-super
• nemotron-3-ultra
• nemotron-3.5-lightning
• nvidia-nemotron
• openfda
• polars
• preact
• pydantic
• python
• redis
• render
• tavily
• typescript