Inspiration
An LLM can write a convincing explanation for a payments discrepancy in a few seconds. In finance that is the dangerous part: a wrong explanation that sounds right gets a case closed and money goes missing. I wanted to see what it takes to use a model for the hard reconciliation cases without ever trusting it as the source of financial truth.
What it does
LedgerGuard reconciles orders, payments, refunds, settlements and bank entries. A deterministic engine and an independent Shadow Ledger resolve everything that can be proved with identifiers and arithmetic: on the frozen holdout that is 54 of 85 cases, with no AI involved. Only the cases they cannot prove go to the model, here NVIDIA Nemotron on Nebius Token Factory. The model returns a structured hypothesis and points at the records that would prove it. An Evidence Gate then checks those records itself: does the refund exist, does it belong to this payment, does the amount restore the ledger. If the proof is not there, the case goes to a human instead of being closed.
Results on Nemotron
On the frozen holdout of 85 payment lifecycles, NVIDIA Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b) on Nebius Token Factory raised accuracy from 89.4% with rules only to 94.1%. All 16 cases it helped auto-resolve were correct, it resolved 4 of 9 unlinked partial refunds with proof, it asked to close 0 of 6 wrong-linkage cases, and nothing was closed wrongly (0 cases, INR 0.00). Every response was valid JSON. Median investigation latency was 2.3 s (p95 3.3 s). Full table: ledgerguard/evaluation/outputs/nemotron_comparison.md in the repo.
How I built it
Python, FastAPI and Pydantic, with all money as integer paise. A seeded synthetic generator produces payment lifecycles with known ground truth and six injected fault classes, split into development and a frozen holdout. The AI step sits behind one interface, investigate(context) -> InvestigationResult, so the model host is a configuration choice. Nemotron is reached through Token Factory's OpenAI-compatible chat completions API. The dashboard is a single HTML page served by the backend.
What is new for this hackathon
I first built LedgerGuard in September 2026 as a solo project. For this hackathon I added a Nebius Token Factory provider that runs the investigator on NVIDIA Nemotron, made the response parser handle the reasoning text that thinking models write before the answer, added a one-command evaluation of Nemotron on the same frozen holdout, added offline tests for the new provider, and added an MIT license. The engine, Shadow Ledger and Evidence Gate are unchanged, which is the point: the model can be swapped and the safety property does not depend on which model it is.
Challenges I ran into
The hard case is a refund whose amount exactly matches a shortfall but which belongs to a different payment. Models tend to call it a partial refund. In the September runs across five investigators, 25 of 30 such attempts would have been closed without the gate and 0 were closed with it. Nemotron 3 Super was the first investigator that never asked to close one. On the provider side, my first choice, Nemotron 3.5 Lightning, returned empty answers in this setup, so the provider fell back automatically to Nemotron 3 Super; every fallback stays on NVIDIA models, and any answer that does not parse goes to human review rather than crashing.
Accomplishments that I'm proud of
The safety property is measured, not claimed: the no-gate ablation shows what would have gone wrong, a test protects the gate, and the Nemotron run closed nothing wrongly.
What I learned
Swapping the model is easy when the model is behind one narrow interface. The hard part is deciding what the model is never allowed to decide.
What's next for LedgerGuard
Find out why Nemotron 3.5 Lightning returns empty answers here and get it working, since it is smaller and faster. Push partial-refund recall above 4 of 9 without loosening the gate. Replace the synthetic settlement data with real settlement files from a test account.
Log in or sign up for Devpost to join the conversation.