Ledger Sentinel

Inspiration

Month-end close is the most repetitive, deadline-driven workflow in finance: reconcile the bank statement against the ledger, hunt for accruals that belong in the period, explain every material account movement, and catch anomalies like duplicate invoices or out-of-period postings. Most of it is mechanical. A small fraction genuinely needs human judgment. Every automation tool I looked at either stops at categorising transactions or asks a human the same question every single month. I wanted an agent that does the mechanical part end to end, escalates only the judgment calls, and never asks the same question twice.

What it does

Ledger Sentinel runs the close for a period on its own: bank reconciliation (exact, fuzzy, one-to-many, Dodo Payments payout matching), accrual detection (received-not-invoiced purchase orders, missing recurring vendor bills), flux analysis against the prior month, and anomaly checks (duplicate invoices, vendor name variants, split invoices under approval limits, out-of-period postings).

It proposes journal entries, each carrying an evidence[] list of the exact source rows it came from. The schema rejects any entry with empty evidence or unbalanced lines, so a hallucinated plug entry is structurally impossible, not just discouraged. Genuine judgment calls go to an exception queue in a React review UI where a human approves, rejects or edits with a mandatory written reason.

Then the part that matters: every human decision is distilled by the model into a narrow policy rule (scope, condition, action, limits). The rule is replayed against all prior months as a regression gate and promoted only if it flips no previously correct decision. Promoted rules apply on the next close. In the demo, January raises 8 exceptions, I resolve the vendor-name ones and distill one rule, February raises 5 with that category gone, March raises 4, with bank auto-match at 100 percent throughout.

How we built it

The whole thing was built inside Agent Orchestrator in one day. I wrote one brief to the orchestrator: read the spec, follow the build plan, review every PR yourself, merge when green. It spawned 14 worker sessions in sequence and in parallel, each in its own git worktree: scaffold, data generator, bank reconciliation, accrual and flux engines, agent core with the learning loop, review UI, eval harness, three eval-driven fix workers, two eval refreshes, and docs. Every component shipped as a focused PR through review and CI (ruff and pytest) before merging.

Stack: Python 3.12, FastAPI, SQLite via SQLAlchemy, pydantic, Vite and React, the Anthropic SDK (with a headless claude CLI fallback) as the single model access point, and Neatlogs tracing with one workflow span per close, one span per engine and one per model call. Deterministic engines do the heavy lifting and are unit tested against fixtures; the model is used only for ambiguity resolution, narrative and rule distillation.

Challenges we ran into

The orchestrator's own review loop caught real bugs: money columns typed as float where SQLite returns Decimal, a test that only passed because engine packages were absent, a period filter missing from accrual matching. The eval harness then found the reconciliation engine under-matching (precision 0.22 in the first run) and the orchestrator spawned fix workers on its own. Later I found that promoted rules never reached the anomaly engine and that distilled rules used approve-style wording the matcher didn't accept, so the learning loop looked like it worked but changed nothing. Fixing that turned the escalation curve from flat to falling. Also, my laptop blue-screened twice mid-build. AO restored the sessions both times.

Accomplishments that we're proud of

Precision 0.97, recall 1.0 and F1 0.985 on 33 planted anomalies across nine categories, bank auto-match 100 percent every month, and an escalation rate that falls month over month because of learned rules, not tuning. The evidence guarantee held everywhere: not a single entry in the system exists without source rows. And the orchestrator reviewed and merged all 14 pull requests itself; I mostly watched the board.

What we learned

Deterministic engines plus a model that only handles ambiguity beats a model that does everything. A learning loop is only real if it is gated: replaying candidate rules against prior months and rejecting any that flip a correct decision is what makes it safe to let the system get more autonomous. And with a good spec and interface contracts, an orchestrator can run a multi-worker build with very little supervision, but the eval harness has to exist early, because it is what tells the orchestrator where to spend its fix workers.

What's next for Ledger Sentinel

Connect real ledgers (QuickBooks, NetSuite, Xero) and real bank feeds instead of the synthetic company, add flux narratives that cite transactions in the reviewer's words, let rules carry expiry and owner metadata for audit, and export a full close package. The learning loop generalises to any recurring finance workflow with human review, invoice processing and cash reporting are the obvious next two.

Built With

• agent-orchestrator
• anthropic
• dodo-payments
• fastapi
• neatlogs
• pydantic
• pytest
• python
• react
• ruff
• sqlalchemy
• sqlite
• typescript
• uv
• vite