Inspiration

Medical bills are hard to read, and LLMs are confidently wrong about money. A model that explains a $551 ER bill as "$551 with a $50 refund" sounds helpful and is dangerous. In a KFF survey, 43% of U.S. adults said they had received a medical or dental bill they thought contained an error (https://www.kff.org/private-insurance/could-consumer-assistance-be-helpful-to-people-facing-medical-debt/). The fix isn't a bigger model. Arithmetic should never be the model's job, and the model's words should be checked before a patient sees them.

Who it's for

Patients reading a claim or an explanation of benefits, and the billing-office and patient-navigation staff who answer their calls. The output is a short explanation of what was billed, what the plan allowed, and what the patient owes and why.

What it does

  1. Deterministic ledger. The claim's lines and plan (deductible, coinsurance, copay, out-of-pocket max) go through a billing engine (billpilot/billing.py) that produces per-line billed / allowed / patient-owes amounts and the total.
  2. Writer: Nemotron 3 Nano on Nebius Token Factory gets only the ledger text and writes a plain-English explanation at temperature 0.
  3. Auditor: Nemotron 3 Super on Nebius Token Factory gets the ledger and the draft and returns strict JSON (ok, problems), flagging any amount, claim or tone that doesn't match the ledger.
  4. Guardrails. Regex checks run last and reject any dollar amount not present in the ledger, plus any draft that doesn't state the total.
  5. Decision. The Nemotron draft is shown only if the auditor says ok and the guardrails pass. Otherwise the patient gets a safe template built from the ledger. It never "retries until it passes": a failed audit is a signal, not noise. For an EOB document, Nemotron 3 Nano also extracts each field with a verbatim quote, and code checks every quote against the document before recomputing the amount owed.

How we built it

  • Two Nemotron models, two jobs. nvidia/nemotron-3-nano-30b-a3b is the cheap, fast writer. nvidia/nemotron-3-super-120b-a12b is the auditor, with the reasoning to catch an inconsistency. That's the "Nano/Super for fast calls… make your credits stretch" pattern Nebius recommends.
  • Token Factory's OpenAI-compatible API. Calls go to https://api.tokenfactory.nebius.com/v1 (nemotron.py). Model IDs are environment variables, so swapping models is a config change.
  • Measured cost and latency per explanation on Token Factory (10 synthetic claims, 2026-10-24; means, output includes reasoning tokens):
Model Role Tokens in/out Latency Cost
Nemotron 3 Nano writer 161 / 944 7.9 s $0.00024
Nemotron 3 Super auditor 254 / 204 1.9 s $0.00026
Total per explanation 9.8 s $0.00050

Design: one claim, one honest answer

The demo page (a replay of the recorded run) is for a patient or a billing-office agent:

  1. Pick a document. The EOB text with every evidence span highlighted; each extracted field links to its line.
  2. Conclusion card. Match, mismatch (stated vs recomputed amount, with the line) or the exact missing fields; no amount is guessed.
  3. Explanation and run cards. The patient text with badges (Nemotron or safe template, auditor verdict, guardrails), plus model IDs, tokens, latency and cost per call.

A billing-office agent can paste the explanation into a patient message with confidence. If the badge says template, the reason tells them what the model got wrong.

Trust by design

  • The model never computes money. Every amount comes from the deterministic ledger.
  • Two independent checks. One is a model-based audit (Nemotron Super) and one is rule-based (guardrails). Both must pass.
  • Fail closed. Malformed auditor JSON, a missing API key, an audit failure or an unsupported amount all lead to the safe template.
  • Transparent output. Each response reports source (nemotron or safe-template), the auditor's verdict and any guardrail violations, so staff can see why.

Accomplishments that we're proud of

Every conclusion shows where it came from. In a live run on Nebius Token Factory, a synthetic EOB goes in, Nemotron 3 Nano extracts each field with a verbatim quote, and code checks every quote against the document (line and character span), recomputes what the patient owes, and compares it with the amount the EOB states. 9/9 runs (3 documents × 3) reached the right outcome: a match at $551.00, a mismatch caught (the EOB says $601.00 on line 10; its own terms give $551.00), and an incomplete document that names exactly what's missing (coinsurance, out-of-pocket maximum) instead of guessing. Median 16.0 s and $0.00087 per document. It explains and flags; a person decides what to do.

8/8 offline tests with mocked Nemotron responses (5 pipeline, 3 document journey):

  • a correct draft is shown;
  • a draft that invents a $50 refund is caught by the guardrails (unsupported_amount:50.00) and replaced;
  • an auditor "not ok" leads to the template;
  • malformed auditor output leads to the template;
  • no model available leads to the template, with the correct $551 total.

Live run on Token Factory over 10 synthetic claims: 10 shown as Nemotron drafts, 0 fallbacks. Auditor stress test (30 calls: a correct text plus an inflated total and an invented $50.00 refund per claim): Nemotron Super flagged 20/20 injected errors and raised 1 false alarm in 10 correct texts (it objected to a line item omitted from the summary). The guardrails missed 1 of 20: a $50.00 "refund" on a claim where $50.00 is a real ledger amount. Super caught it, which is why both checks are required. Total spend for all live runs today: about $0.03.

What we learned

Arithmetic should never be the model's job, and the model's words should be checked before a patient sees them.

Challenges

  • Getting the auditor to return strict JSON reliably. Anything else is treated as a failure.
  • Deciding that a failed audit must fall back rather than retry.
  • Keeping the writer from adding "helpful" amounts such as refunds, estimates or discounts that aren't in the ledger.

Limitations

  • Synthetic claims and text EOBs only; no PDF parsing yet.
  • English output only.
  • Not medical, legal or billing advice.
  • The guardrails check amounts and the presence of the total, not every phrase.
  • The demo page replays the recorded Token Factory run (no live calls), so the API key is never exposed.

What's next

  • EOB PDF ingestion.
  • Spanish and Portuguese output (the auditor checks the translation against the same ledger).
  • Testing with billing-office staff.
  • A batch mode on Token Factory for a whole day's statements.

Run it

python3 test_pipeline.py && python3 test_journey.py   # 8/8 offline, no key
NEBIUS_API_KEY=... python3 journey.py docs/eob_er_mismatch.txt

Team

Solo build by Igor Eduardo (Austin, TX): design, code, evaluation and video. Founder of Cortexa Social, Inc.; Antler Founder Residency (F26, Austin). AI coding assistants were used during development.

Video note: the narration is a synthetic text-to-speech voice (en-US-AndrewNeural); visuals are screenshots of the recorded Token Factory run from 24 Oct 2026. All claims and EOBs are synthetic.

Built With

  • nebius-token-factory
  • nemotron-3-nano
  • nemotron-3-super
  • nvidia-nemotron
  • python
Share this project:

Updates

Submission history