Model X-Ray

The debugger for financial models. An AI agent that finds the errors hiding in Excel workbooks and proves every fix. Designed for everyone who needs their numbers to be right.

Comment

Inspiration

Every big financial decision starts in a spreadsheet, and one wrong cell is enough to sink it. In 2012 a JPMorgan risk model divided by a sum instead of an average, and the "London Whale" trades lost $6.2 billion. In 1995 a single missing minus sign turned a $1.3 billion capital loss at Fidelity Magellan into a capital gain.

Real models have thousands of formulas across dozens of tabs, and nobody has time to trace every one by hand. AI looks like the obvious reviewer, but language models are built to reason, not to calculate. They can explain a number convincingly and still get it wrong. A financial model doesn't need a plausible answer. It needs a proven one. That gap is what we wanted to close.

What it does

Model X-Ray is the debugger for financial models. You upload an Excel workbook (or press Try the demo), and the agent checks it the way an experienced financial analyst would:

  • Finds the errors on its own. No question needed. It reads every sheet, labels every line and period, and lists every suspect cell. All suspects are investigated at the same time.
  • Diagnoses each one. It traces the dependency graph, finds the root cause and tells a real error from a deliberate choice, such as a value typed on purpose and explained in a note.
  • Proves every fix. The fix is applied to a sandbox copy and the whole workbook is recalculated. Then a second agent, the Verifier, tries to break it. Every objection it raises is re-checked by the engine.
  • Explains it in plain language, with the cell, the cause, the formula as it is, the tested formula and every cell the fix changes, shown in a viewer that keeps the workbook's own design.
  • Waits for you. You approve, edit or reject each fix, and you can ask the agent about any error or fix. Approved fixes go into a corrected copy with a change log. Your original file never changes.

It covers seven error types (typed values, wrong references, flipped signs, incomplete ranges, wrong periods, logic errors and unusual inputs) and 52 Excel functions, including XLOOKUP, SUMIFS, NPV, IRR and OFFSET, plus cross-sheet references, defined names and circular references.

How we built it

The core idea is a strict split: Nemotron reasons, the engine calculates. No number in an answer is guessed by a language model.

  • Our own calculation engine (Python) parses every formula, builds the dependency graph and recalculates the workbook with Excel's semantics, including iterative calculation for circular references.
  • A LangGraph agent with ten steps: load the workbook, map the model, locate, trace, diagnose, test, verify (with a bounded retry loop back to diagnose), explain, human review and apply the fix.
  • Three NVIDIA Nemotron models on Nebius Token Factory, each chosen for its job:
    • Nemotron 3.5 Super is the Investigator and the Verifier.
    • Nemotron 3 Super maps every line and period of the workbook.
    • Nemotron 3.5 Lightning explains each error and answers the reviewer's questions.
  • Deterministic rules first, Nemotron where judgement is needed. Rules propose suspects (pattern breaks, incomplete totals, double counts). Nemotron judges each candidate, reviews the model's logic line by line three times and keeps what a majority finds.
  • A FastAPI backend and a React frontend on Vercel, with live progress streamed to the page. The review state travels in the browser, so the server keeps nothing between requests.

Challenges we ran into

  • Language models and arithmetic. Early versions let the model state numbers, and it was confidently wrong. We moved every number into the engine and made the Verifier's objections checkable claims (a cell and a value) that the engine recalculates before they count.
  • Telling an error from a choice. Rules alone flagged every value set on purpose: 12 false alarms out of 12. Teaching the agent to read notes and context brought that to zero.
  • Reasoning budgets. Nemotron 3.5 Super sometimes spent its whole output budget thinking and never reached the tool call. We now ask once more with thinking turned off.
  • Errors that look like answers. Under load, some calls came back as HTTP 200 with an error message as the reply. We read and retry every answer instead of trusting the status code, and wait 5, 15 and 30 seconds on overload.
  • Several errors in one file. Finding all errors at once was easy to say and hard to do. Each fix must be tested alone, then approved one after another on the copy that already holds the earlier fixes.

Accomplishments that we're proud of

  • 98% of planted errors fixed (82 of 84) on our synthetic set, against 86% for rules alone.
  • Zero false alarms on values set on purpose, where rules alone failed 12 of 12.
  • 18 of 20 planted errors fixed on five Damodaran valuation models never used for tuning, with all 20 located and no false alarm on the clean originals.
  • About a minute and 1.4 cents per error checked (medians on unseen models).
  • A live demo anyone can try: a European power purchase agreement model with 2,400 formulas and three planted errors, all found in one run.

What we learned

  • Separating reasoning from calculation is what makes an AI answer trustworthy. Nemotron is excellent at judging what a formula should be. The engine is what proves it.
  • An adversarial second agent pays off, but only when its objections can be checked. A Verifier that may say "no" without evidence just adds noise.
  • Model choice matters per step. Moving the reasoning steps from Nemotron 3 Super to 3.5 Super was a one-line change on Token Factory, and fixed errors on an unseen SaaS model went from 0 of 5 to 4 of 5, at the same price.
  • Measure everything. Planting errors with known correct answers let us score each step and catch regressions we would never have seen by eye.

What's next for Model X-Ray

  • Google Sheets. The sandbox already sits behind the same interface as Excel; the next step is wiring it into the app.
  • More of Excel. Links to other workbooks, Data Tables, R1C1 formulas and numbers stored as text.
  • Sharper period checks, including fiscal-year labels such as "FY2030", and whole-row fixes for wrong references.
  • Memory per reviewer, so a value you marked as deliberate is never flagged again, across sessions.
  • Persistence and team review with a shared database, so a review can be paused, shared and resumed.
  • Real-world error benchmarks beyond planted errors, and a comparison with Nemotron Ultra as the Verifier.

Built With

Share this project:

Updates

Submission history