Tieout

An agent that only works the invoices that broke. It investigates each one across the ERP, email and a vendor portal with no API, then learns the policy from one approval so the next clears itself.

Comment

Inspiration

Accounts payable automation is a solved problem — for the invoices that match. Every AP tool on the market races to straight-through-process the clean ones.

Then we found the number nobody builds for. The invoices that need manual rework are about 12.5% of the ledger, they take 45–60 minutes each to resolve, and they eat 60–70% of an AP team's working time. One vendor calls exceptions "the Achilles heel of AP automation".

(Those figures were gathered on 2026-09-03 from zamp.ai, peakflo.co and IOFM via Rossum, and are recorded with their publishers and that date in RESEARCH.md. They are secondary citations — the industry's stated figures, not our measurements. Every other number in this project came out of a real run and the command that produces it is printed next to it.)

So everyone automated the invoices that were never the problem.

And the reason the broken ones stay manual is not that the decision is hard. It is that the evidence lives in three places — the ERP, the email thread, and a supplier portal with no API that a person signs into by hand every morning. Gathering it is the work. Deciding takes seconds.

You cannot automate the judgement. You can automate the investigation. And you can learn the judgement — once.

What it does

Tieout ignores the clean invoices and works only the ones that broke. Four beats:

1. Investigate. A short-shipped invoice. Tieout runs a real three-way match (purchase order vs goods receipt vs invoice), then walks three sources: it reads the ERP, reads the vendor's email thread, and signs into the vendor portal in a real browser to pull the delivery note — because supplier portals do not have APIs. An evidence pack appears: every fact with its source, the exact URL to fetch it again, the time it was read, whether code or the model read it, and a screenshot for anything read off the portal. Then a proposed decision, with a confidence.

2. Decide once. A human clicks Approve. A policy is born, stamped with the approver's name and the date.

3. Clears itself. The next short-ship from that supplier resolves with no human at all, citing that rule and that person. It does not sign in again — the portal session was saved.

4. Refuses. One exception where nothing lines up. Tieout says "I am not confident. Here is everywhere I looked. You decide." Refusing is a first-class outcome, not an error.

   match ──▶ investigate ──▶ decide ──▶ escalate ──▶ learn ──┐
     ▲                                                       │
     └───────────────── the next one of that kind ◀──────────┘
                         clears itself, citing the rule

The measurement

Two short-shipped invoices from the same supplier, same world, run twice from a cold reset:

Before the loop After one approval
Human touches 0 (nothing decided yet) 1
Waiting on a person 2 0
Auto-cleared by a learned rule 0 1
Human touches avoided 0 1

Two exceptions both needed a person. Now one of them does not — and it is not skipped, it is decided: short-paid to the delivered quantity, citing SHORT-SHIP-01 v1 and Chris, Controller, with its own evidence pack behind it. Every later short-ship from that supplier inside the tolerance costs zero touches.

The counter is computed from the store, never hard-coded.

How we built it

The whole build ran inside AO, one session per phase of the plan, each in its own git worktree and branch — eleven of them by the end, plus an orchestrator session. Because the phases map onto separate modules, two sessions almost never touched the same file even though the backend is a single Python project.

world → engine → api → desk → harden

Two things the worktree model actually bought us, rather than the tooling story:

  • The desk session rebuilt its TypeScript types against the merged engine/models.py. The earlier skeleton had guessed them from the plan and got several wrong. That is one session reading another session's merged output, which is the point.
  • The harden session found two bugs only repetition finds. Driving the same four beats through the screen five times in a row surfaced both: the counter strip could read all zeros after Reset, because the desk asked for counters in parallel with the queue — and it is the queue read that runs the match and fills the store. And the counters lagged a whole beat behind the screen, because a run was marked finished before its refetch landed. One pass would have shown neither.

Stack. Python 3.13, FastAPI + SSE, Playwright for the portal, Next.js 16 + shadcn/ui for the desk. The model is glm-4-7-flash over TensorMux, reached as a plain OpenAI-compatible HTTP call, so any provider is a base-URL swap.

Where the loop lives. Two files own everything that matters:

  • engine/policy.py — the only file that creates, versions, matches and applies rules. A rule is applied by code, never by a model. The model may suggest a name and a tolerance; policy.py clamps every number before storage — never narrower than the case it was learned from, never wider than the class ceiling — and always pins the rule to that supplier. The rule notes on itself when it narrowed the model's draft. Nothing is ever deleted: a contradicting decision produces a new version, and the old one stays on the record marked superseded.
  • engine/evidence.py — the only writer of the audit trail. A fact that did not come through this file does not exist. A portal fact whose screenshot is not on disk is rejected. A fact dated outside the open fiscal window is rejected. Empty statements are rejected.

The confidence is not a model output. Each exception class has a written checklist; each line passes or it does not; the confidence is the weight that passed. A judge can read the checklist and predict the number. Below CONFIDENCE_FLOOR = 0.70, or with a required check failed, Tieout refuses.

What the model is allowed to do is three named jobs in one file, engine/model.py: turn email text into checkable facts, draft the rationale for a decision the code has already made, and draft a candidate rule for policy.py to clamp. Matching, evidence collection, rule application and the confidence gate are deterministic code. That is why "auto-cleared, citing SHORT-SHIP-01 v1, approved by Chris, Controller" is reproducible on a rerun instead of a story the model told.

Run it with no key at all and the demo still works — the model's three jobs degrade to prose the code writes itself, and the trail says read by code instead of naming the model. We verified this. Identical either way: the exception count, the human touches, the auto-clear, the refusal, and the rule's id, version, condition, action, approver and citations. Different: the rule's human-readable name and its rationale paragraph, and the evidence count (12 facts instead of 16, because the model splits a vendor email into up to three checkable claims where the code records it verbatim as one). Nothing the loop depends on is a model call.

One approval is not an unbounded approval. engine/authority.py holds the delegation-of-authority matrix every AP department already has — AP Clerk £1,000, Controller £10,000, CFO no limit. A decision above the approver's limit is blocked, not recorded. And a learned rule inherits the ceiling of whoever approved it, enforced inside policy.covers, so a Controller cannot create a rule that later clears a 50,000 invoice on its own. That is segregation of duties, enforced rather than hoped for.

Challenges we ran into

The model disappeared mid-build. The model we had wired up was removed by the inference provider partway through the weekend. Because every LLM call already went through one file with a three-line env config, the swap to glm-4-7-flash was a config change, not a refactor. The boundary paid for itself the same day we drew it.

The replacement returned nothing. glm-4-7-flash is a reasoning model: it puts its thinking in a separate field and, at a low token ceiling, spends the entire budget there and returns empty content. It looked like an API failure. The fix was to turn thinking off explicitly in the request — which also took a call from 19.7s / 1,912 tokens down to 2.9s / 150.

The CLI was running dead code. An editable install was pointing at an old AO worktree, so tieout demo had been executing a stale copy of the engine for several sessions. Every run before we caught it was measuring the wrong thing. Reinstalling from the main repo fixed it — and it is the reason we now print the counters from the store on every run rather than trusting a previous result.

The model guessed the year. On the very first test call it dated a fact 2024. A silently wrong year is exactly the bad evidence that must never reach a human, so evidence.py now rejects any fact outside the open fiscal window. That rule exists because of one bug on day one.

Merging eleven branches. Real conflicts, including one where a feature session had built on components that the main branch had since replaced. Resolved by taking the newer side and re-wiring the feature on top, rather than merging both and hoping.

"How does a browser log in on a server with no screen?" The honest answer has three tiers, and we built the middle one — after one successful sign-in, sources/portal.py saves the browser context with Playwright's storage_state, and the next investigation loads it and walks straight in. That is why beat 3 prints "reused the saved VendorLink session — no sign-in needed" where beat 1 printed "signed in as [email protected]". "Logs in once" is demonstrable here, not hypothetical.

Accomplishments that we're proud of

  • The loop is asserted, not demonstrated. test_policy_loop.py approves E1 once and asserts that E2 auto-clears citing SHORT-SHIP-01 v1 and Chris, Controller. 56 tests, and the Python ones run offline — conftest.py empties the model key, so every reproducible part of the loop is proved with no model in the room.
  • The engine cannot cheat. test_the_engine_cannot_read_the_seed scans every engine file to prove the match never reads the world's answer key.
  • Byte-identical across five consecutive cold runs — the facts, the checklist, the confidence, the rule, the citation and every counter. The one thing that varies is the model's rationale paragraph, and it is labelled with who wrote it.
  • The agent signs into a real portal with a real form and a real session cookie, and screenshots what it read, because that is the part of AP work nobody has an API for.
  • Refusing is built, not bolted on. Beat four exists because an agent that never says "I don't know" is not one a finance team can use.

What we learned

Draw the model boundary on day one. One file, three named jobs, everything else deterministic. It survived a provider deleting our model, and it is the only reason the demo is reproducible.

A self-improving loop is mostly about what the learning is not allowed to do. The interesting code in policy.py is the clamping, the per-supplier pinning, the required-check gate and the versioning — not the learning. An agent that widens its own rules is not a feature, it is an incident.

Repetition finds a different class of bug than tests do. Both desk bugs needed the same five beats run five times in a row. Neither would have failed a unit test.

Parallel agent sessions need module boundaries, not just branches. Where the plan's phases mapped cleanly onto folders, eleven sessions merged almost without conflict. Where two sessions drifted onto the same component, we paid for it in a merge.

What's next for Tieout

  • Real ERP connectors. sources/erp.py already parses JSON into its own DTOs and never imports the fake world — a test enforces that — so NetSuite and SAP are new source adapters, not a rewrite.
  • Remote browser handoff for SSO and 2FA. The browser runs on the server and streams its screen into the user's own tab; they sign in, do 2FA on their phone, and the cookie is created where the agent needs it. Same shape as a bank's "connect your account" flow. That is a week, not a weekend.
  • More exception classes. Three today (short_ship, price_variance, missing_po). A fourth means a fourth written checklist — deliberately, so the confidence stays predictable.
  • Rules that widen carefully. Policies are learned per supplier today. Widening to a category or a whole ledger should require its own approval at the right authority level, which is exactly what authority.py is already for.
  • Approvers as identities. The approver is a name in a text field today. In a real deployment it is an SSO identity, and the rule should carry it — which also makes the delegation matrix read from the company's own policy instead of a constant.

Built With

Share this project:

Updates

Submission history