-
-
Qwen plans over messy financial inputs; a deterministic gate is the only path to a write; governed write-back to a real Odoo 19 ERP.
-
Gate off: the same Qwen model posts 5 wrong entries to a real ledger, salaries and cost-of-goods misbooked, all balanced and Posted.
-
A large entry is not blocked, it is held and escalated to a human; a CFO sign-off releases it and the same gate then approves and writes.
-
A balanced entry booked to the wrong account passes a trial balance; reconciliation to the source catches it and the gate refuses it.
-
A correct entry passes the eight exact gate checks, is approved, and is written to Odoo as one real, posted account.move.
-
Qwen3-VL-Plus reads a scanned invoice. The 4,500 the gate reconciles against was never typed by a human; wrong account still gets rejected.
-
Qwen drives the ERP write over MCP and posts move_id 3. Told to inflate the amount, the server refuses: the signed token no longer matches.
-
The backend runs on Alibaba Cloud ECS, provisioned from code via the ECS and VPC OpenAPIs. The instance is live and serving over HTTPS.
Inspiration
I work close to accounting and ERP systems, and I kept seeing the same tempting idea: point a capable LLM straight at the general ledger and let it post the month-end journal entries. The problem is that a ledger is a system of record, and a single wrong entry posted to it is not a small mistake. It is an audit adjustment, with review, rework, and restatement risk attached. On a real accounting-workflow benchmark the best models are still wrong on roughly one task in six, and when an agent is allowed to write behind a single weak control the false-success rate climbs sharply. So I built LedgerPilot around one decision I made on day one and never walked back: the model is allowed to reason and propose, and it is never allowed to write. Every write passes a deterministic gate first.
What it does
The clearest way to explain LedgerPilot is a controlled experiment. Take the same Qwen model and the same 39 month-end close tasks, and post everything it drafts to a real Odoo ledger twice: once with the gate off, once with it on. Gate off, five wrong journal entries land in the ledger, and they are all still there under refs prefixed NG-WRONG: salaries paid out of accounts receivable, cost-of-goods booked to receivables and revenue. Every one balances and passes a trial balance. Gate on, same model, same tasks: 0. The model did not get better; the ledger did. That is the whole product in one number, and it is impact in a general ledger rather than a percentage of my own test set.
Underneath: LedgerPilot reads the messy inputs a finance team drowns in every month (bank statements, supplier invoices) and a Qwen planner drafts the journal entries. Those proposals never touch the ledger directly. They pass through a deterministic validation gate that runs eight exact, auditable checks: double-entry balance to the cent (and that the entry actually moves value), account validity, no account on both sides, positive amounts, period lock, segregation of duties, approval thresholds (with a human-in-the-loop escalation above a limit), and reconciliation against the source document. That last check is the one a trial balance cannot do: it catches a perfectly balanced entry posted to the wrong account or for the wrong amount, which is exactly the mistake a generative model makes. The gate fails closed: any failing check other than the human-approval threshold rejects. Only an entry that passes every check is bound to an HMAC-signed approval token and written to Odoo, idempotently on retry.
Two measured numbers back the counterfactual. On an offline synthetic stress-test of 204 cases (12 scenarios across 14 error classes), the gate wrote 0 wrong entries of 36 approved (at most 8.33% at 95% confidence), caught all 168 seeded errors, and wrongly blocked 0 clean controls. On live Qwen output across 39 tasks, run on the Alibaba Cloud ECS box: eight model mistakes across two models, eight caught, zero wrong writes (Wilson 95% CI at most 9.18% and 10.72%). The weaker model made seven of the eight, and the ledger stayed clean. The honestly disclosed limit is that the posting policy is per-document, so a permitted-but-wrong posting would surface as a nonzero rate; in these runs every error fell outside the permitted set and was caught, which is what makes the metric falsifiable rather than zero by construction.
How I built it
The core idea is a clean split between generative and deterministic work. Qwen on Alibaba Cloud Model Studio is the generative layer: Qwen3.7-Max plans the entries with function calling, calling a lookup_accounts tool to discover valid account codes rather than being handed the chart, so account selection is grounded in real data. The deterministic layer is the gate: pure, side-effect-free Python with Decimal-only money (it refuses to construct money from a float), so every verdict is reproducible and auditable. Approvals are carried by HMAC tokens bound to a hash of the exact entry, including its amounts, accounts, narration, and the preparer and approver, so neither the numbers nor the audit trail can be altered after approval without invalidating the token.
The part I am proudest of is the MCP integration. The gate is exposed as an MCP server, attached to Qwen through Model Studio's Responses API as an SSE tool. This is what makes the MCP use meaningful rather than decorative: the model is the caller of the write tool, not the authority. validate_write is read-only; execute_approved_write re-runs the full gate and verifies the token server-side before it touches Odoo. In the demo, Qwen posts a real account.move, and when I instruct it to inflate the amount first, the content hash changes, the token fails, and the server refuses. The model holds the pen and still cannot forge the cheque.
The backend runs on Alibaba Cloud ECS, and the infrastructure is itself code: scripts/deploy_ecs.py calls the ECS and VPC OpenAPIs to create the key pair, security group, VPC, vSwitch and instance, and tears it down with --destroy. The agent, the gate, the MCP server, and the write-back all execute on that box, which also serves the gate's web UI over HTTPS at https://ledgerpilot.jonathanandrei.com. The whole thing is covered by 79 tests that run in under a second, including regression tests for every hole a review of the gate turned up (an empty entry, a same-account wash entry, tampering with the narration or the approver after signing).
Challenges I ran into
The hardest problem was measuring honestly. My first evaluation handed the gate the correct answer for each task, which made the false-write rate zero by construction: a meaningless, circular number. I tore it out and rebuilt it so the gate validates each proposal against a per-document posting policy that is independent of the single correct answer, which means a plausible-but-wrong posting can actually pass and be counted. The second challenge was the semantic error a balance check cannot see: an entry that balances perfectly but is booked to the wrong account. Reconciliation against the source document is what closes that gap.
The third was adversarial: I ran a review that tried to break the gate, and it did. An entry with zero lines "balanced" (0 == 0) and slipped through; a same-account wash entry did too; and because the signed hash originally covered only the amounts, a valid token still verified after the narration or the approver's name was rewritten. On the MCP path, where a language model chooses its own tool arguments, a model could even disable the reconciliation check by omitting the source document. Each of those is now a fix and a regression test: the gate requires an entry that moves value, the hash covers the full content and provenance, the config refuses to run against a live ERP with the development signing key, and the network-facing MCP gate refuses to write without evidence. The test suite went from 53 to 79. Finding those before a judge did is the difference between a demo and a control.
The last was infrastructure timing: Alibaba Cloud identity verification gates model access, so I built and validated everything credential-free first, then deployed to ECS and re-ran the whole pipeline there.
Accomplishments I'm proud of
A deterministic trust boundary that is the only path to a write, and a measured false-write rate reported with a confidence bound rather than a vendor-style accuracy percentage. A reconciliation check that catches the balanced-but-misposted entries a trial balance passes. Governed, idempotent, tamper-evident write-back to a real ERP with a human-in-the-loop gate. Each gate check mapped to a named control (SOX 404, COSO, PCAOB AS 2201, ACFE segregation of duties). And an honest scope section that says plainly what is a synthetic stress-test versus a live measurement versus a future prospective study.
What I learned
That the hard part of putting an LLM next to a system of record is not the model, it is the write governance and the honesty of the measurement. A deterministic-plus-generative architecture lets each layer do what it is good at: the model reasons over messy, ambiguous input, and exact rules make the irreversible decision. I also learned to be disciplined about statistics: zero observed failures is not zero failures, so I report a Wilson or Rule-of-Three bound instead of a bare zero. And I got a real feel for the Qwen Cloud surface, from the OpenAI-compatible endpoint to function calling to MCP over the Responses API.
What's next for LedgerPilot
Deriving the posting policy from the ERP's own chart-of-accounts structure rather than a per-document list, and escalating genuinely ambiguous account choices to a human rather than accepting any permitted account. Per-line reconciliation against the source document's net and tax split (today the gate validates the total and the permitted account set, not the per-line VAT breakdown). Reversal entries and a persisted audit log and review queue, which are designed into the topology but not yet built. A live prospective study against real closes with a gold standard, to turn the synthetic bound into a field measurement. And scaling from single entries to a full automated close cycle against a live ERP.
Log in or sign up for Devpost to join the conversation.