Confidence Gate

Inspiration

Every redaction pipeline has the same shape: a model reads a document, decides what is sensitive, and something removes it. The demos always work, because the demo document is the one the pipeline was built against.

Two things go wrong in production, and neither one raises an error.

It removes nothing. Redaction matches literal text. If the model reports a value that is not in the document as written, the match finds nothing, removes nothing, and the API returns 200. The operator sees success and releases a document that still contains everything.

It removes too much. One literal can occur in several places. Redact one and you destroy the others.

Both failures are silent, and both are decisions about a regulated document that no person ever made.

What it does

Confidence Gate sits between the agent and the irreversible call.

1. extract - Nutrient Data Extraction returns values with per-field citations
2. ground - classify every field by where it came from, not by its score
3. plan - translate "redact these fields" into "remove these strings"
4. refuse - collateral damage, ungrounded values, empty values, with reasons
5. review - print a one-time token; record the plan under a hash
6. approve - a person, with the token, on the record, reason required if blocked
7. apply - Nutrient Processor, createRedactions + applyRedactions
8. verify - re-read the output as text: did exactly the authorised strings go?

The decision is made on grounding, not on the score. Nutrient's guide describes the extraction confidence as "a relative and uncalibrated signal". That is the correct description, and it means a threshold over that number is a threshold over nothing. A value the model invented is blocked at 0.99, while a value demonstrably in the document passes at 0.4.

There is also a mechanical reason, not just a philosophical one: an ungrounded value cannot be redacted at all. There is no string to match. Blocking it is the only honest answer.

How we built it

Python 3.11, no third-party dependencies - multipart/form-data is assembled by hand so there is no dependency tree for a reviewer to audit.

Nutrient DWS does the load-bearing work in four places:

Endpoint	Used for
POST /extraction/extract	schema-shaped values with citations - the match label is the entire decision procedure
POST /extraction/parse	plain text of both documents, for verification
POST /build (createRedactions + applyRedactions)	the irreversible step, always issued together
POST /build (parts: html)	building the sample fixture, so it is reproducible rather than a checked-in binary

The gate has three properties: an approval binds to the SHA-256 of the document and of the exact strings to remove; the review token is printed once to the terminal and never written to disk; every step appends to a hash-chained ledger. The ledger stores the SHA-256 of each removed string, never the string - a compliance record that itself holds what you redacted is not one.

Challenges we ran into

The API returns coordinates. Redaction accepts none. Extract reports a bounding box for every value; the redaction endpoint takes only text, regex, or presets. Translating a location into a string is lossy in both directions, and that gap is the whole problem this project addresses.

Our own verification was wrong, and it took a measurement to find out. We first verified by running the schema extraction again on the output and seeing which fields came back empty. On a real redacted file it reported three untouched fields as destroyed - a home address, an application date, an effective date. Parsing the file as plain text showed all three intact. Extraction is layout-sensitive: put black boxes on a page and unrelated fields stop resolving. The same fragility runs in the other direction, so a pipeline verifying this way can also report a value as removed while it is still legible. We rewrote verification to compare text presence and pinned the exact failing scenario in tests/test_verify.py.

A field being blank is not the model hallucinating. Our first audit called an empty date_of_birth an invented value. It was the opposite - the form is blank there and the model correctly declined to fill it in. Conflating the two would have cost the tool its credibility on the cases that are real hallucinations, so they are now separate findings.

Accomplishments that we're proud of

The sample document defeats a confidence threshold, and we can show it. In the onboarding form, the settlement account number, the employer's public company registration number, and an internal reference are all the same eight digits, three sections apart:

field	match	confidence	value
company_number	id_match	0.95	10462298
account_number	id_match	0.95	10462298
internal_reference	id_match	0.95	10462298

Every one is grounded correctly at a high score. A score threshold passes all three and destroys the wrong ones. Confidence Gate refuses, and says why.

40 tests, none of which need a key or a network. Each is built from a hand-written API response, deliberately: a decision that only holds when a live service happens to agree is not one you can put in front of an auditor.

The honest limits are in the README, not buried. id_match proves a string is in the document, not that it was attached to the right field. The out-of-band token falls to anything that can read the operator's terminal. Verification cannot see text rendered as an image. A tool that overstates itself is worse than no tool in a regulated environment.

What we learned

That the interesting failures in document AI are not the model being wrong. They are the seams between components - where a location becomes a string, where a success code stands in for evidence, where a layout change looks like data loss. Every real defect we found this build was in a seam, and each one only surfaced when we ran the thing against live documents instead of fixtures.

And that a verification step you have not tried to break is not a verification step. Ours was wrong for three fields, on the first real document we pointed it at.

What's next

Coordinate-addressed redaction would close the gap properly - the extract endpoint already returns bounding boxes, so if createRedactions accepted them, the collateral-damage class of failure would disappear rather than being routed to a human. Until then, routing it is the right answer.

Beyond that: batch review for document sets, a webhook so approvals can happen out of band, and using Classify to select the schema automatically rather than being handed one.

Built With

• document-ai
• nutrient-dws
• python
• redaction
• rest-api
• sha-256