ExceptionZero

• 

• 

• 

• 

• 

• 

Inspiration

Automation didn't eliminate this work. It concentrated it.

Every organisation that moves money runs some kind of automation over it, and every one of them still has a queue of exceptions — payments that failed, cash that arrived with no reference, amounts that don't reconcile. That queue is small precisely because the automation is good. What's sitting in it is the residue: the cases the rules were never able to cover.

And the residue is the expensive part. A clean payment clears in seconds. One of these takes a person twenty minutes — pull the invoice, check the customer record, look at what happened the last three times this counterparty paid, work out which system is wrong, fix it, verify it. A small share of the volume, a disproportionate share of the labour, and the place where cash sits uncollected and regulatory exposure quietly accumulates.

There is software for exception queues. It sorts them, scores them, routes them. It gets a case to a person faster. It does not resolve the case — because resolving one means making a judgment call on incomplete evidence, and that has never been something a rules engine could do.

So the interesting question isn't whether an agent can work faster than a person. It's whether an agent can be trusted to make that judgment call — and, more importantly, whether it can be trusted to recognise when it shouldn't.

That second half is the whole project. An agent that resolves everything is not useful here; it's dangerous. The one that earns its place is the one that resolves what the evidence supports, refuses what it doesn't, and tells you which is which.

ExceptionZero was built to see whether that's achievable with real guarantees — not a prompt asking a model to be careful, but identity enforced by IAM, a decision boundary that is deterministic code, and evidence rules a hallucinating agent cannot talk its way past.

What it does

ExceptionZero resolves workflow exceptions autonomously, and escalates the ones it shouldn't touch.

A representative run against real Gemini 3.5 Flash on 20 payment exceptions: 11 resolved autonomously, 8 escalated, 1 quarantined as a prompt-injection attempt. Roughly 40% resolved without a human, and every escalation carries a stated reason.

Most of those escalations are correct by design. An invalid account number or a sanctions hit should never be auto-resolved, because fixing it requires contacting a human being. The interesting case is the one the fleet wanted to resolve and didn't:

The payment amount of EUR 44,704.73 is significantly less than the invoice amount of EUR 81,281.32 for INV-20232, and there is no evidence explaining this large partial payment.

Confidence landed below the floor and the value above the ceiling. Two independent conditions, either sufficient to stop it. Escalation is a success state, not a failure.

Escalated cases don't disappear. They wait in a review queue with the reason they stopped and what would have to change to clear it. A Cloud Scheduler sweeper re-examines them and re-opens the ones whose blocker has cleared — in testing, four cases closed 23 days after they were first deferred.

How we built it

Ten agents, three tiers of authority.

Triage classifies. A coordinator dispatches specialists — chosen for the exception type, not all of them every time, which cuts specialist queries by 45%. Four specialists (invoice, counterparty, history, precedent) run concurrently, each under its own service account, each scoped to exactly one BigQuery table. Diagnosis reasons over what they found. A deterministic risk gate decides whether to act. Execution holds the only write credential and no decision authority. Verification confirms or rolls back.

Identity is enforced, not declared. Every tool call runs under credentials impersonated for that agent's service account. The diagnosis agent's SA holds aiplatform.user and cloudtrace.agent and no data role at all — so it cannot query the estate even if it tried. The deployed service proves this live at GET /identity, which has each agent attempt a real BigQuery read under its own identity and reports the result.

Routing is dynamic where it's safe and fixed where it isn't. When diagnosis can't establish something, it names what's missing; the coordinator dispatches those specialists and the case is reconsidered. That decision is made at runtime by the agent's own assessment. But the control plane is fixed: the guards run in order, the gate is deterministic and always last. An agent that could both decide and execute is the failure mode the design exists to prevent.

Seven guardrails, all mechanical: Pydantic-validated handoffs; two-layer evidence-citation enforcement; a deterministic risk gate; a four-turn loop guard; idempotency keys; compensating actions written before execution; and a circuit breaker that halts the fleet after three verification failures in twenty cases.

Five fault modes ship with the project — hallucinated citations, phantom entities, infinite loops, verification failures, forced overconfidence — injectable from the UI, because a well-behaved model can't demonstrate recovery.

The engine is domain-neutral, and that's testable. EZ_DOMAIN=supply_chain runs the identical fleet against receiving exceptions with different thresholds. fleet_core.py does not contain the word "payment".

Technologies used

Gemini 3.5 Flash via Vertex AI · GenAI SDK and Google ADK (LlmAgent, output_schema, before_model_callback) · Cloud Run · Pub/Sub · Firestore (agent registry, deferred-case memory) · BigQuery (the data estate) · Cloud Trace with OpenTelemetry · Model Armor (regional template, prompt-injection and sensitive-data filters) · Cloud Scheduler (the sweeper) · IAM service-account impersonation · Python 3.12 · FastAPI · Pydantic

Gemma is wired as an optional pre-classifier for triage (EZ_GEMMA_MODEL / EZ_GEMMA_ENDPOINT), routing the cheap high-volume label to the small model. It is not provisioned in the hosted demo — a dedicated GPU endpoint would cost several hundred dollars over the judging window — so the fleet falls back to Gemini automatically.

Data sources

All data is synthetic and generated by a seeded script in this repository (generate_dataset.py), so a judge can reproduce the exact estate with one command. No real payment data, no customer data, no employer data.

The estate: 339 exceptions across 8 types, 700 invoices, 60 counterparties across 8 countries, and ~780 payment-history records. Crucially, the generator also produces the corroborating evidence that makes each exception resolvable — a duplicate submission has its original settled payment on file, an amount mismatch has prior short-pays by the same fee that were accepted.

Three cases are engineered deliberately: a large unexplained shortfall the fleet must refuse, a prompt injection in a payment memo field, and a reference to an invoice that does not exist.

Challenges we ran into

Untrusted text reaches the model after ingestion. A payment memo is text a stranger typed, and it flows into agent context like any other field. Screening at message ingress isn't enough, because a tool can retrieve untrusted content later in the same turn. Moving screening to the ADK before_model_callback — between the agent and Gemini, over the fully assembled prompt — closes that gap.

A guardrail that silently stops guarding. A naive substring check for MATCH_FOUND also matched NO_MATCH_FOUND, so every clean filter result registered as a hit and the fleet quarantined 100% of its input. The service returned 200s and well-formed JSON throughout.

Making declared identity real. Setting a service account on a registry record is a label. Binding it to execution took google.auth.impersonated_credentials on every tool call, plus serviceAccountTokenCreator grants on nine accounts.

Trace context doesn't cross threads. The specialist fan-out was real and completely invisible — each specialist became its own root trace, so Cloud Trace showed a linear pipeline. Explicitly attaching the parent context inside each worker thread fixed it.

Accomplishments we're proud of

Every claim in this submission is verifiable in a Google Cloud console rather than taken on faith. The agent registry in Firestore. The identity model in IAM — and provable live at /identity. The reasoning chains in Cloud Trace, showing three specialists in parallel and an adaptive re-dispatch. Model Armor templates in Security. The sweeper in Cloud Scheduler.

And the thing the project is actually about: an agent that computes a €36,576 shortfall, states it cannot explain it, and stops.

What we learned

A rigorous reasoner is a data-quality test. Stub agents resolved 40% of cases because the type-to-action mapping was hardcoded. Real Gemini resolved zero — and was right to. Every escalation pointed at a flaw in the synthetic estate: duplicates with no original on file, invoices marked open that had already settled twice. The model refused to assert what the evidence didn't support. Every jump in resolution rate came from fixing the data, never from loosening the agents.

Reversibility can't be self-reported. The diagnosis agent decided that voiding a duplicate was irreversible and blocked its own resolutions. The fix wasn't a better prompt — it was removing the question from the model. An action is reversible if and only if a compensating action is defined for it. Letting an agent describe the consequences of its own proposal is the same failure the gate exists to prevent.

Confidence saturates without anchors. Early runs returned 1.00 on every case, including a 45% unexplained shortfall. Numeric anchors plus a required unexplained field — where anything listed forces confidence below the floor — produced scores the gate can actually use.

Well-behaved models make bad demos. The planted hallucination bait never fired, because the model correctly reported the invoice as absent rather than fabricating one. Deliberate fault injection turned out to be both more honest and more convincing than waiting for a failure that shouldn't happen.

And the finding that cost the most time: in an agent fleet, a component that does nothing and a component that works look identical from the outside. Model Armor fell back silently to a local detector. The registry fell back silently to memory. A scheduler pointed at an endpoint that didn't exist. The specialist fan-out ran but emitted no spans. Every one returned a clean, plausible, well-formed response. Not one was caught by an error — each was caught by checking a number against what it should have been. That is why the guards emit spans and the traces carry attributes: not for the judges, but so the failure and the success stop looking the same.

What's next

Tightening the coordinator's read scope, which currently holds broader BigQuery access than it needs because the service bootstraps under it. A reviewer interface on top of the queue, so a human can act on an escalation rather than only read it. And a third domain connector, since adding one costs a taxonomy, an action list, and a policy table.

---

Try it yourself

Demo	https://exceptionzero-vkh44dcwiq-uc.a.run.app
Prove the identity model	GET /identity — the diagnosis agent attempting a read and being refused
Published agent catalog	GET /registry
Human review queue	GET /queue
Run the fleet	POST /run with {"limit":20,"workers":8}
Inject a fault	add "inject":"hallucination"

Built With

• bigquery
• cloud-run
• cloud-scheduler
• cloud-trace
• fastapi
• firestore
• gemini
• google-adk
• google-cloud
• iam
• model-armor
• opentelemetry
• pub-sub
• pydantic
• python
• vertex-ai