Intact

• 

  workflow

Inspiration

Discharge instructions get handed to a patient at the worst possible moment. Tired, stressed, often reading a second language, and usually at a reading level well above their own. The fix is a nurse sitting down to explain, which needs an expert and rarely happens properly.

The obvious solution is to hand the paper to an LLM and ask for plain language. We tried that first. It works until it does not, and the ways it fails are the ways that hurt people. It alters a dose. It quietly drops a warning sign. Worst of all, it adds reassurance the document never contained, and a patient who reads "this is a routine recovery, nothing to worry about" may not act on a symptom they were told to watch for.

So the interesting problem was not "can AI simplify this". It was "can you prove it did not change anything".

What it does

INTACT takes a discharge document and returns it rewritten for the patient, in a chosen language and reading level, with a guarantee attached: every dose, every date, every drug name and every warning is provably unchanged, and nothing has been added.

When it cannot do that safely, it says so and falls back to a plainer rendering rather than publishing something it cannot verify.

How we built it

The design decision everything rests on: the rewriter never sees the source document. It only sees an extracted fact ledger.

A rewriter holding the original text inherits every clinical implication in it, plus everything the model already knows about the condition, and it fills perceived gaps helpfully. Giving it a closed set of facts removes the material it would hallucinate from. That is a constraint in the architecture, not a request in a prompt.

Then two gates decide whether anything is published, and neither one is a model call:

1. Preservation. Every quantity is parsed with pint and compared by magnitude within its own dimensionality, so 500 mg and 0.5 g match while 500 mg and 50 mg do not, and 5 mg never matches 5 ml. Drug names are matched strictly with no brand to generic substitution. Warnings are checked for survival and separately for retained urgency.
2. Additions. Any claim that cannot be traced to a ledger fact is stripped from the output, not merely flagged. A report saying "this sentence was invented", attached to a document that still contains the sentence, is not a safety mechanism.

On failure the pipeline rewrites once under a stricter constraint, and failing that renders the ledger through templates. The system degrades toward less fluent output, never toward less accurate output.

Results

Against a live single prompt (Llama 3.3 70B, given a prompt that explicitly tells it not to change doses, not to add advice, and not to omit warnings):

                            WORKFLOW      SINGLE PROMPT
mean fact preservation        100.0%             93.4%
dangerous failures                 0                 5
invented claims                    0                17
documents safe to publish        3/3               0/3

The single prompt dropped two urgent warning signs, lost two inhaler strengths, invented a number that appears nowhere in the source, and added 17 claims the clinician never wrote.

Separately, injected corruption testing, which needs no API key at all:

dangerous corruptions intercepted : 20/20 (100.0%)
all corruptions intercepted       : 26/26 (100.0%)
tests                             : 31 passed

Nine corruption types across three documents. Decimal shifts, unit swaps, altered frequencies, shortened durations, dropped medications, dropped warnings, softened urgency, invented advice.

Challenges we ran into

The check we were most confident in had a hole in it. Our adversarial harness tests "warning softened", where "go to hospital immediately" becomes "go to hospital when convenient". It scored 0 out of 3. Every content word survives, so the overlap check passes. No number changes, so preservation passes. The patient is told what to look for and not that it needs acting on now.

We added an urgency check. It still failed, because "emergency" was in our list of urgency words, and "emergency department" names a place, not a timeframe. The softened sentence sailed through. The real fix needed location words removed and explicit detection of softening phrases.

We were also unfair to our own baseline, twice, and had to fix it. Our first comparison counted extraction gaps as fabrications, so the baseline looked unsafe for reasons that were actually our extractor's misses. Then our addition check flagged "You have had a laparoscopic appendectomy", which is true and in the document, just outside our ledger's schema. Both had to be separated out before the comparison meant anything.

Accomplishments we are proud of

The guarantee is enforced by deterministic code, so anyone can clone the repo with no API key and reproduce every safety number we claim.

We also report the numbers that do not flatter us. Ledger recall is 83.6% offline, and we say so, with a worked example of a fact the extractor missed.

What we learned

Prompts are requests, not guarantees. The baseline prompt explicitly said do not change doses and do not omit warnings, and the model did both anyway. What actually held was withholding the source document from the rewriter and checking the output with code.

We also learned that a test suite which only ever confirms your design is not testing anything. The single most useful hour was the one where our own harness found a hole we had not imagined.

What's next

Ledger recall is the weak link, and no amount of downstream verification repairs a fact that was never extracted. That is where the work goes next.

Beyond that: more languages with native speaker review, a wider corpus, and the same architecture applied to other high stakes documents such as tenancy agreements and government benefit letters. The mechanism is domain general.

Built With

• featherless-ai-(llama-3.3-70b)
• httpx
• matplotlib
• openai
• pint
• pydantic
• pytest
• python