SigCheck-PILLSPARK

Inspiration

When we first read PillSpark's challenge, we thought the problem was fairly straightforward: take a faxed prescription and turn it into something a pharmacist doesn't have to type out manually.

Our first instinct was OCR. There are already models that can read documents pretty well, so we thought the main challenge would be extracting information accurately. But while looking through prescription examples, something kept bothering us. What if a model reads a drug name incorrectly, but the name it comes up with is also a real drug? A database would recognize it. Another model might even agree with it. Nothing would necessarily look wrong.

We started questioning whether focusing entirely on better extraction was enough.

That's where SigCheck came from. We wanted to build something that doesn't just read prescriptions but also checks those readings using information and rules outside the AI models.

What it does

SigCheck takes prescription faxes and turns them into structured drafts that pharmacists can review without manually retyping everything.

We built two parts around this idea: a verification engine and a pharmacist workflow app.

The verification engine reads prescriptions using Tesseract OCR and two AI models from different vendors. It compares their readings and runs independent checks against medication references, prescriber identifiers, prescription rules, and known fax-related problems.

The workflow app handles the steps around that process. Pharmacy staff can upload a prescription or capture one using a phone. The app extracts its information, shows the original image alongside the fields, and identifies what needs to happen next.

We separate cases that need transcription verification from those requiring prescriber clarification or those blocked by missing information or suspicious instructions. For example, if a prescription contains an ambiguous strength such as 20.0 mg, the pharmacist cannot simply confirm the uncertain reading and approve it. The case waits for an unambiguous clarification from the prescriber.

We also built a review queue, source highlighting, pharmacist approval with recorded decisions, an audit trail, and downloadable PDF and JSON draft orders.

The two components currently run alongside each other, with their own extraction and checking pipelines. Connecting the full verification engine directly to the live workflow is one of our next steps.

How we built it

We used Python and FastAPI for the backend and web application.

For the verification engine, we used Tesseract OCR to extract text and word locations. We also used GPT-5.5 and Mistral Large 3 through Nicrron's API to independently interpret the prescription and return structured JSON.

The engine checks medication names, strengths, and dosage forms against RxNorm, uses openFDA for controlled-substance information, and includes rules for NPI and DEA checksums, prescription restrictions, ambiguous notation, and fax-quality problems.

For the live workflow application, we used Apple Vision OCR for text recognition and source highlighting, alongside Claude Sonnet 5 through Nicrron for structured extraction.

We used PRISM by Block Convey for tracing, prompt-injection guardrails, output evaluation, and knowledge-base lookups. The workflow app queries PRISM's Knowledge Base using kb_search, checks the retrieved reference against the exact catalog entry, and displays PRISM trajectory scores. We also masked patient and prescriber identifiers before sending telemetry.

For testing, we generated fictional prescriptions using Pillow, adding blur, skew, noise, streaks, and other fax-like effects. We kept answer keys for the generated images so we could measure actual extraction and verification results.

Challenges we ran into

One of the most interesting cases involved a fax where the left edge was cut off. Both AI models read escitalopram as citalopram. Both are real medications, so a simple database lookup wouldn't have caught the mistake. Our page-truncation check flagged it.

In another test, one model read a noisy warfarin 5 mg prescription as 6 mg. Again, both strengths exist. Our noise check caught the issue and sent it for review.

But we also found that being too cautious creates problems of its own.

In one clean prescription, Tesseract inserted a space into an NPI, triggering our cut-off-page check even though the models had read the number correctly. We could see how to improve the rule, but we'd already frozen our checks before running the evaluation, so we left the result unchanged.

We also had some unexpected results with PRISM. Its input guardrail didn't catch the prompt-injection instructions planted in our test faxes, although our local detector did. The evaluator also identified inconsistencies in two verification traces.

Integrating the verification engine with the live pharmacist workflow was another challenge. We built and tested both components, but connecting all 38 engine checks to the live app would have required more time than we had.

Accomplishments that we're proud of

We were especially happy to see our independent checks catch mistakes that model agreement and ordinary drug lookups would have missed.

We froze our verification rules before generating a separate held-out test set. On that small synthetic evaluation, 235 out of 235 fields that passed our checks matched the answer keys, and all 10 planted traps were detected. We also recorded three false alarms among ten clean faxes.

Alongside the verification engine, we built a working pharmacist review application rather than stopping at a command-line demo. It supports live prescription uploads, source highlighting, clarification requests, pharmacist-attested approval, PDF and JSON export, and a recorded history of decisions.

The workflow app also passed its own synthetic evaluation, catching 5 out of 5 planted traps, with 48 out of 48 passed fields matching the expected values and no invented values in that test.

We ran 85 Python tests and three UI tests, all passing.

These are small synthetic evaluations, not evidence that SigCheck is ready for clinical use. But having measurable results, including false alarms and known limitations, was important to us.

What we learned

We started out thinking that having two AI models read the same prescription would help us catch mistakes. It did in some cases, but we also found examples where both models agreed and were still wrong.

That made us realize how much the condition of the original fax matters. A drug database can confirm that a medication exists, but it can't tell us whether the first few letters were cut off or whether a speckle changed one digit into another.

The false alarms taught us something too. It's easy to keep adding checks, but every unnecessary warning means more work for the pharmacist. We need to measure both the errors we catch and the correct prescriptions we interrupt.

We also learned that building individual components is different from connecting them into one usable workflow. We had a detailed verification engine and a separate live intake application, and bringing the full engine into that workflow is still something we want to finish.

By the end, we were thinking less about how accurately a model could read a prescription in isolation and more about what the entire system should do when that reading might be wrong.

What's next for SigCheck-PILLSPARK

Our first priority is to connect the full two-model verification engine and its 38 checks directly to the live pharmacist workflow, so every newly uploaded prescription can go through the same evaluation process.

We also want to improve review prioritization so potentially incorrect medication names, strengths, and directions receive attention before less critical issues.

Finally, we'd want to test SigCheck on a wider variety of prescription images, including real handwriting and different fax conditions, using appropriately authorized data and feedback from pharmacists.

We started with the idea of reducing manual prescription entry. What we'd like to keep working on is making sure the time saved doesn't come at the cost of mistakes that are harder to notice.