PillSpark

Inspiration

Pharmacies still receive prescriptions as faxes. Around nine billion of them move through US healthcare every year and at the receiving end a person may barely be able to ready, yet retypes it by hand into a pharmacy system.

The obvious fix is OCR and that fix is already well funded. What struck me is that the obvious fix is also the dangerous one. A vision model handed a smudged drug name does not return failure provenance. It returns the most probable drug name because that is what next-token prediction does. Hydralazine becomes hydroxyzine. Celebrex becomes Cerebyx. A decimal drops out of the scan and 5 mg becomes 50 mg. Every one of those comes back fluent, confident, and wrong, and the pair of drugs you get confused between are the exact look-alike/sound-alike pairs that medication safety groups already track as a leading source of dispensing error.

So for this project, I didn't want to build an extractor and focus on what an extractor is missing. started: a system that knows which fields it does not know and is right about that. In the ever going world of agent automation, I believe this is cruicial.

What it does

PillSpark turns faxed prescription images into structured orders covering patient, drug, dose, directions and prescriber, and flags every uncertain field for a pharmacist instead of filling it in.

The important word is flags. When PillSpark is not confident, the field comes back empty rather than guessed. A withheld value is a feature, not a failure.

Each field carries the evidence behind it:

• A confidence score built from real signals, not the model's own opinion of itself. We use disagreement between independent extractors, external registry validation, and plausibility checks, because a model's self-reported confidence does not correlate with whether it is correct.
• Pixel provenance. Every field points back to the region of the image it came from, so a pharmacist verifies against the actual crop rather than retyping from scratch.
• A full decision trail. Which models produced the value, which validators ran and what they returned, which threshold was applied, and why the field was accepted or flagged.

The headline metric is not accuracy. It is autonomy rate at a fixed error ceiling: what share of fields can be auto-accepted while holding critical-field error below a stated bound, with everything else routed to a human. A system at high accuracy that cannot tell you which fields are wrong is worth nothing to a pharmacy. We report a coverage and risk curve instead of a single number.

How we built it

Pipeline. A fax image goes through multi-pass extraction, then a consensus layer, then validators, then triage into verified, needs-review, unreadable or missing. Output is a versioned order schema where every field is a full object carrying value, confidence, status, candidates, provenance and reasons.

Ensemble disagreement as the confidence signal. We route the same crop through several different vision models through the Nicrron gateway and treat divergence on drug, strength or sig as an automatic flag. The gateway is not there for convenience. Multi-provider access is what makes disagreement measurable in the first place, which is the whole basis of our uncertainty estimate.

Validation against external truth. Drug and strength existence against RxNorm and NDC, manufactured-strength and max-daily-dose plausibility, prescriber NPI against NPPES, DEA checksum on controlled substances. A field that reads perfectly cleanly but fails "this strength is not manufactured" still gets flagged.

Thresholds derived from measurement. We built an evaluation harness over a labeled synthetic corpus, including a controlled degradation set of 32 single-factor variants isolating blur, skew, low DPI and fax compression artifacts. The per-field abstention thresholds the runtime enforces come out of that measured curve and ship as a versioned artifact, not as numbers somebody guessed.

Data boundary. Synthetic and de-identified images only, everywhere, through every API. Prompt caching disabled. Identifier extraction kept separate from clinical field extraction, so patient demographics and drug data never travel together.

Challenges we ran into

Our first evaluation was measuring itself. Early on, mock extraction perturbed ground-truth JSON rather than reading pixels. It produced beautiful numbers. Perfect precision, clean coverage curves, zero critical-field errors. All of it circular. We had evaluated a simulator of our pipeline rather than the pipeline.

What gave it away was a detail that should not have been possible: the degraded image slice scored better than the clean one. Blur and skew and compression should make flag recall harder, not easier. That inversion told us the perturbation was modelling label noise instead of image degradation, and those two things are exactly what we claim to tell apart. We threw out the numbers and rebuilt on real vision calls over real image bytes. Accuracy dropped hard, which was the correct outcome and the first honest result we had.

Confidence is not a number a model can hand you. Asking for a 0 to 1 score gives you something fluent and uninformative. Everything useful had to be assembled from outside the model: cross-model agreement, registry hits, arithmetic reconciliation between quantity, sig and days supply.

Vendor limits we could not engineer around. The Nicrron docs warn that automatic failover can route a request to a provider other than the one you asked for, and document no per-call failover-off setting. Explicit model pinning is not the same as a guarantee. Rather than pretend otherwise, we recorded it as a known limitation and kept all upstream input synthetic, which is the actual guard.

Scoping against funded competition. Tennr has raised roughly $161M for healthcare fax automation. Asepha, TJM Labs, Pharmesol and Parachute Health all ship document intake in this space. We had to find the part that was genuinely open rather than rebuild theirs, and that search is what pushed us toward calibration.

Accomplishments that we're proud of

To not report numbers that we could not rightfully back. The easy version of this project reports the mock figures and wins on a slide. We labeled them invalid, kept them visible next to the real ones, and rebuilt. Every number in this submission carries the extraction mode that produced it.

A working abstention layer, not an extraction demo. Fields come back empty when the system is unsure, and we can show the coverage and risk tradeoff behind that choice rather

Built With

• claude
• codex
• nicrron
• prism