INSPIRATION
I work at a B2B telecom consultancy that audits phone bills for mid‑sized companies. Every month analysts open invoice PDFs one by one compare each line against the signed contract and against what the account consumed in cycles and write up whatever does not add up.
It is slow it does not. It is exactly the kind of work that gets skipped when the month is busy. Which is when the money leaks. And outside a consultancy it is worse: most companies simply pay the bill because it arrived. No company can audit its phone bill. This agent does that job end to end.
WHAT IT DOES
One invoice PDF in. A defensible dispute letter out. Nobody in the loop.
Reads the PDF. Gemini 3.5 Flash multimodal extraction into a schema with a deterministic repair loop when Pydantic rejects the output.
Cross‑checks it. Every line against the contracted plan and against the accounts billing cycles. Five rules in three families, Python, Decimal arithmetic running concurrently under a ParallelAgent.
Decides what to do. It separates what the carrier owes back from what the customer's wasting drafts the carrier dispute letter and a separate customer summary verifies every figure in both and escalates what it is not sure about to a human.
The load‑bearing claim: no money figure ever comes out of the LLM. It is enforced in three layers than promised in a prompt.
Structural: the rule engine is Python with Decimal; no module under rules/ imports an LLM client.
In the tool signatures: no auditor tool accepts a value as an argument. A test asserts this with inspect.signature.
In the generated prose: amount_guard checks every number in the letter against the set the engine actually computed. Invented one? The model is told which. Rewrites it. Insisted? The dispute is stored as blocked never as a draft somebody might later mistake for reviewed.
It refuses than guesses. An unknown carrier is refused, not read with another carriers layout hints. An invoice whose printed carrier contradicts the one you named is turned away before an extraction is paid for. And with no contract on file it says it could not audit the invoice. It never calls it clean because silence from a rule that did not run is not evidence that the bill's correct.
RESULTS
Measured on 15 invoices across 4 accounts and 2 visually distinct carrier layouts (Brazilian and American). Every number is reproducible offline.
Field accuracy: 99.55 %. 1781/1789 Values
Schema validity: 15/15, with 0 repair rounds needed
Anomalies found: 5/5. Recall 100 %
positives: 0. Including a clean control account
Recovered total: 1,036.10 vs 1,036.10 ground truth to the cent
Carrier identified from the printed letterhead: 16/16, with no model call
Test suite: 203 tests, offline, no credentials, about one second
99.55 % extraction accuracy still yields 100 % detection and exact money: the eight residual errors are quantity/unit_amount on a fee line whose columns the American template never prints. They do not move a finding.
HOW I BUILT IT
Gemini 3.5 Flash via Vertex AI. PDF transcription, contract transcription and the single judgment step.
Google ADK 2.7.1. A SequentialAgent pipeline: intake, invoice_extractor, auditor, dispute_writer with the rule families under a ParallelAgent inside the auditor.
Cloud Run (us-central1). Dedicated service account with object‑level access to one bucket rather than project editor; min‑instances 1 max 2 concurrency 8 so the cost has an explicit ceiling.
Firestore. Contracts, canonical invoices keyed by content hash, findings, disputes, review queue.
Cloud Storage. The two generated documents, published as ADK artifacts with a 30‑day lifecycle.
Cloud Trace. With ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false after I verified in the real spans that the default was exporting transcribed invoice content.
Only three of the eleven pipeline stages are LlmAgents. Intake and the rule families are deliberately not: a model call whose only job is to invoke a function is tokens spent on nothing. It dresses deterministic work up as reasoning.
OTHER DATA SOURCES
No real customer data. The dataset is 15 PDFs and 4 contracts generated with ReportLab from a declarative spec committed to the repo with ground truth.
It also ships one invoice and its signed contract built independently from public material about how carriers here actually bill. A cycle that does not start on the first taxes itemised because Lei 12.741/2012 requires it identifiers in the shape a real bill prints them. Nobody designing it knew which rules would read it. Auditing it the first time surfaced three defects at once and all three were positives; none of the 15 synthetic invoices could have caught any of them. It is committed at data/independent/. Locked green by an offline test as the false‑positive control with the generator taken out of the loop.
Learnings
Almost everything I learned came from testing the deployed service with documents nobody prepared knowing what would be checked. Not from reading my code.
A health certificate is worse than no answer. With no contract on file three of five rules are. The findings list comes back empty. And the prompt told the model to say the invoice "looks clean". A brand‑new account the first path any evaluator walks got that certificate over a bill that had an overcharge. Silence from a rule that never ran is not evidence.
A model asked to transcribe invented clauses. Transcribing a 3‑plan contract it filed 6. Helpfully re‑adding each one under the abbreviated name the invoice prints. The contract is the baseline every figure is computed against so it now contained terms nobody signed. Fixed with a validator, not a prompt.
A correct transcription can still be an audit. A Brazilian invoice came out 149.42 out of balance because ICMS, PIS, COFINS FUST and FUNTTEL are computed inside the price and itemised only because the law requires it. The schema now declares the tax regime per carrier.
Testing against my generator only proves my generator is consistent. Three defects surfaced at once the time I audited an invoice built from an independent source.
The money parser was losing the minus sign four ways. The regex that strips currency noise keeps an ASCII hyphen so U+2212 (what a properly typeset PDF prints) an en dash, an em dash and (50,00) all came back as positive fifty. A credit read as a charge moves the invoice total by twice its value and nothing downstream could catch it: the sign is gone before the value is ever a Decimal and the guard on the letter compares it against what the engine computed, not against the page. Found by going looking for it. The parser had no tests all.
Refusing at the door only works if the stages downstream ask. Sibling stages of a SequentialAgent cannot be skipped from outside. The extractor was reading the attachment on its own. Extracting anyway right past the refusal.
WHAT'S
More carrier profiles (two ship today and anything else is refused rather than misread); more adversarial documents, in the regression suite; and persisting the audited PDF so an audit can be re‑checked against the document that produced it.
Built With
- cloud?run-cloud?storage
- cloud?trace
- firestore
- gemini?3.5?flash
- google?adk
- pydantic
- python
- vertex?ai