Close Controller

Inspiration

Reconciliation is a long tail of failure modes wearing the same costume — two rows that should have been one. A controller closing a month opens two spreadsheets, works down a hundred and eighty bank lines against a hundred and seventy-five ledger entries, and spends three days on it.

Most of that work is mechanical. The value sits in the handful of rows that are genuinely wrong: a duplicate payment, an invoice billed above what was actually received. Those are precisely the rows that three days of manual matching cause a person to miss.

Existing automation fails in one of two directions. Rules-only tools match the easy cases and tip everything else into an undifferentiated "unmatched" pile. LLM-only tools match everything, explain nothing, and cost too much per transaction to run at real volume. We did not want to assume the second claim, so we measured it — the baseline column below is that measurement.

What it does

Close Controller reconciles a bank statement against a general ledger, books what it is certain about, and hands a reviewer the rest as a typed queue with the evidence already attached.

Four tiers, ordered by certainty:

Tier	What it does	Cost
1 · Deterministic rules	Exact reference, same-day amount, fuzzy vendor, subset-sum for partial payments, duplicate detection	Free
2 · Gemini	Only what the rules could not place. Batched, structured output, every returned id validated	~1 API call per close
3 · Confidence gate	≥ 0.90 posts unattended · below routes to a human · < 0.45 is never proposed	—
4 · Typed exceptions	Nine categories, each with a suggested one-click action and its evidence	—

Every posting passes a policy engine that refuses unbalanced entries, duplicate payments, postings into a closed period, and unattended posts above the approval limit. It runs on the agent's own postings and on human approvals — a reviewer approving a duplicate at the end of a long close is exactly the case it exists for. A blocked entry is persisted with the rule that stopped it, never silently dropped: a refusal you cannot point at afterwards is indistinguishable from a bug.

Every decision, agent and human alike, lands in an append-only audit trail in the same shape, exportable as CSV. An auditor can reconstruct the month without opening the app.

Measured, not asserted

The benchmark ships with ground truth — 196 labelled rows across 14 real reconciliation failure modes: timing differences, partial payments, netted bank fees, duplicate payments, FX variance, reversals, three-way over-billing, payments above the approval limit, vendor trading names, free-text ledger memos, processor payouts covering several receivables, and human-transposed references. Ground truth is written only by the fixture generator, never by the agent.

npm run eval -- --baseline disables the rule tier entirely and routes every transaction through the model. That column exists so the tiering claim can be falsified rather than taken on trust.

Metric	Baseline (LLM-only)	Close Controller
Match F1	92.1%	98.3%
Exception recall	90.0%	100%
False auto-posts	1	0
Cost per 1,000 transactions	$0.539	$0.052
Wall time	54.8 s	9.2 s

The LLM-only baseline booked one wrong journal entry with no human involved. The tiered pipeline booked none — at a tenth of the cost and six times faster.

63 unit tests cover the matcher and the policy engine, written against each function's stated contract rather than against the fixture. The eval harness exits non-zero on any false auto-post, so that failure stops a release rather than depending on someone noticing.

How we built it

Everything ran through Agent Orchestrator. The architecture was designed for parallel agents from the first commit: interface contracts were written and committed before any implementation, with file ownership assigned per worker, so agents could run concurrently on disjoint files.

Three AO sessions ran at once on isolated worktrees:

Session	Branch	Output
`syndicatebymaximor-1`	`ao/…-1/csv-import`	CSV ingestion — upload, column detection, date and amount parsing, preview
`syndicatebymaximor-3`	`ao/…-3/tests`	63-test suite over normalize, rules and guardrails
`syndicatebymaximor-4`	`ao/…-4/audit-export`	Decision log, exception and journal CSV exports

Three agents, 28 minutes, 3,691 lines. Sequentially the same work took 61 minutes of agent time. Every session is recorded in docs/AO_LOG.md against the merge commit it produced, so the claim is checkable with git log --graph --all.

Stack: Next.js · TypeScript · SQLite · Gemini API (@google/genai) · Neatlogs for tracing. Money is integer cents everywhere; no float touches a monetary value.

Challenges we ran into

The rule tier was too good, which made the architecture a lie. On the first fixture the deterministic tier scored precision 1.0 and recall 1.0 — the model tier had nothing to do. An architecture that looks good in a diagram and proves nothing. Rather than weaken the rules, we made the data honest: the fixture gained cases unreachable by string normalisation — vendors trading under abbreviated names, ledger memos describing the spend rather than the payee, processor payouts covering several receivables, and references a human had mistyped. The rules correctly leave all of them alone.

A silent failure that only a real run exposed. Gemini 3.x rejects thinkingBudget: 0 with INVALID_ARGUMENT. Our pipeline swallowed the 400s exactly as designed — the close never failed — so the only symptom was a model tier that quietly contributed nothing while every number still looked plausible. MINIMAL is the correct way to ask for no deliberation. We would never have caught this from types or tests; it took running it against the real API.

The model's explanations were useless. It returned reasoning like btx_0004 for -$1450.00 matches gle_0003 for -$1450.00 — restating what the reviewer is already looking at. We rewrote the prompt to demand the semantic link in the reviewer's terms. It now returns "ACME CLOUD SVCS on the bank statement matches Acme Cloud Services in the ledger", and with that change it also started catching a transposed reference it had previously left unmatched.

The model tier was invisible. Its matches post above the confidence gate, so they never reach the review queue — the only trace was a count on the dashboard. A claim that a tier earned its place has to be inspectable, so we added a Matches view listing every pairing with its tier, confidence, signal tag and reason.

Parallelism is not free. The three AO sessions owned disjoint directories, which is why they could run at once, but two collisions still needed hands: two sessions both added a script to package.json, and one built a page main had already gained. Interface contracts prevent conflicts inside a module; they do not stop two agents reaching for the same shared file.

What we learned

A deterministic spine is what makes an LLM's contribution auditable. In finance a reviewer has to explain why the system acted. "The orchestrator chose to investigate" is not something a controller can put in front of an auditor.
Cost tracks ambiguity, not volume. Because the model only sees the residue, spend per close is a function of how much the rules failed to place — a number you can measure and drive down. One API call per close, at this size.
A benchmark you wrote yourself proves less than it looks. The baseline column and the unit tests exist because we wrote both the fixture and the matcher, and that needed an answer.
Agents catch things you missed. One session replaced our hard-coded reporting period with detectPeriod() — without it, every posting from an imported statement would have failed CLOSED_PERIOD the moment a judge tried their own data.

What's next

Direct bank and ledger connectors so a close needs no CSV export at all; learned matching rules promoted from repeated human decisions, so the free tier grows and the model tier shrinks over time; and multi-entity consolidation.