Verdict

Inspiration

Germany is moving business invoicing from PDF to structured e-invoices (EN 16931). Every company has had to be able to receive them since January 2025, and sending them becomes mandatory in 2027 or 2028 depending on size. Public-sector buyers already require the XRechnung format today. Most small businesses still produce PDFs.

The obvious shortcut is to let a language model read the PDF and write the XML. For an invoice, "probably correct" is not good enough. A model that guesses a missing IBAN, copies the buyer's city into the seller's address, or adds up a total on its own can produce a file that passes validation and is still wrong. We wanted an agent that is useful precisely because it refuses to guess.

What it does

Drop a text-based PDF invoice. Every run ends with one of three stamps:

• Verified: every field is quoted from the page or computed, and the official KoSIT validator accepted the XRechnung. Download it as XML, as a ZUGFeRD PDF/A-3 with the same XML embedded, and with a hash-chained audit log.
• Returned: a fact the XRechnung needs is not on the document, such as the Leitweg-ID or the IBAN. Verdict asks for exactly that fact, records the answer as user input, and validates again.
• Rejected: the validator says no and nothing on the page can fix it. Verdict explains the rule in plain English, with web sources, and makes nothing up.

A live routing slip shows each station as it signs off, with timing and cost. A gallery of nine recorded runs on synthetic invoices replays without spending model budget.

How we built it

• Extraction: nvidia/nemotron-3-super-120b-a12b on Nebius Token Factory reads the PDF's text layer and returns JSON under a strict JSON schema. Every value carries a verbatim quote and page number.
• Deterministic core (TypeScript): checks that each quote is really on the page, normalizes units, countries, currency and dates to code lists, applies shape and role rules, and computes all line amounts, VAT and totals with decimal arithmetic. A per-line cross-check requires quantity × price to equal the printed line amount. A deterministic builder writes the CII XML. The model never writes XML and never does math.
• Judge: the official KoSIT validator 1.6.3 with the XRechnung 3.0.2 configuration runs in a small Java service. "Valid" means only ACCEPTABLE from that validator.
• Repair loop: on a rejection, nvidia/Nemotron-3-Ultra-550b-a55b proposes JSON patches, each with a quote from the page. A patch guard drops patches without evidence, on computed fields, or taken from the other party's address block. At most three rounds, and only while the error count goes down.
• Explanations: rules outside a hand-written table of 29 are explained at run time by Nemotron 3 Super, using Tavily search results as sources. Each explanation is cached per rule and shown next to the validator's own wording.
• ZUGFeRD: Mustang renders a PDF/A-3 from the validated XML and embeds it byte for byte. The file is only handed out after both KoSIT and Mustang accept it.
• Audit: every step is appended to a SHA-256 hash chain (RFC 8785 canonical JSON). Model events store model ID, prompt hash, tokens and cost, never invoice contents.
• Hosting: one container (Next.js app plus the validator on localhost) on Google Cloud Run, scaling to zero, with a per-IP rate limit and a daily model budget. Nothing is stored.

A typical invoice costs about 0.3 to 0.4 US cents in model calls and takes about 7 seconds.

Challenges we ran into

• No vision model. Token Factory offers no Nemotron vision model, so we built on the PDF text layer and refuse scanned PDFs with a clear message instead of degrading silently.
• Plausible but wrong values. The worst failure is a valid XRechnung with wrong content, because the validator cannot see it. A misread line item ID once became a quantity and produced a valid file with a wrong total. That led to the per-line cross-check. Later, the model filled a missing seller city with the buyer's city (same postcode) and put a whole printed line into the buyer reference. Each case became a deterministic rule in assembly and in the patch guard, with a test.
• Choosing the right model per task. We tried Nemotron 3 Nano for explanations first. It confused the payee with the buyer, so explanations use Super.
• Honest measurement. LLM output varies between runs, so every change was compared document by document, not only by totals. A scoring bug in our own harness once reported wrong totals that were not wrong; we fixed the scorer rather than the claim.

Accomplishments that we're proud of

• In every document Verdict output as valid, across all sets including 20 held-out invoices, the grand total and amount due match the ground truth.
• Development sets (synthetic PDFs rendered from the public KoSIT test suite): clean 42 of 45 valid, noisy layouts 25 of 30, and 27 of 28 deliberately defective documents ended exactly as a correct pipeline must (verified, asked, or rejected).
• Holdout, run once and reported unchanged: 8 of 12 and 6 of 8 valid, with no invented amounts. Documents that were not valid ended as a question, not as a guess.
• The whole pipeline, benchmark and reports are open source and reproducible.

What we learned

The model is very good at reading and quoting, and the useful work is everything around it: refusing values that are not on the page, computing instead of trusting printed numbers, and letting an independent validator have the last word. Every bug that mattered was a plausible value, not a crash.

What's next for Verdict

• A vision or OCR model for scanned invoices, measured against the same benchmark before it is switched on.
• A "data in, XRechnung out" API for billing systems that already hold structured invoice data.
• More invoice types (credit notes, allowances, prepayments).

Built With

• decimal.js
• docker
• google-cloud-run
• java
• kosit-validator
• mustangproject
• nebius-token-factory
• next.js
• nvidia-nemotron
• pdf.js
• react
• tailwindcss
• tavily
• typescript
• vitest
• zod