Dispute Fleet

The AI proposes. The system governs. And the human remains accountable.

Comment

Inspiration

My sister moved money out of her investment account and into her bank account. She needed it for an urgent bill. That was the whole reason for the withdrawal.

It never arrived.

She called the investment company. They told her the money had left them. She called her bank. They told her they had not received it. The investment company said to take it up with her bank. The bank said to be patient.

That went on for over two weeks. She called repeatedly. She cried on the phone. In the end she borrowed money from someone else to pay the bill she had made the withdrawal for in the first place. The money eventually arrived, two weeks late, by which point the thing it was meant for had already been solved another way.

What stayed with me was not that it took two weeks. It was that at no point in those two weeks was anyone accountable for it. Two institutions each held a record, the records disagreed, and the person in the middle had no way to make either of them act.

That is the exact shape of the problem this product is built around. One of the demo cases in Dispute Fleet is a customer who says a payment failed, sitting beside a bank record that says the transaction SETTLED. The console puts both claims on the same screen, as structured data, side by side. Not because software resolves a contradiction on its own. Someone has to be able to see it before anyone can be held to it.

On the other side of those phone calls is usually one person. A small Nigerian fintech might have a single dispute officer handling hundreds of complaints, working against deadlines set by the Central Bank of Nigeria: 3 days to acknowledge a dispute, 14 days to resolve it, with financial penalties for breaching either.

AI is obviously useful here. It can read a complaint, classify it, compare it against the bank's transaction record, and propose an outcome in seconds instead of days.

But nobody sensible lets a language model decide who gets refunded ₦1.9 million.

That tension is the whole project. The question is not whether AI can help with disputes. Clearly it can. The real question is how do you build a system where the AI genuinely helps, and where you can prove to a regulator, an auditor, or a nervous head of ops exactly what it was never allowed to do? My background is in RBAC, moderation tooling, and admin systems. In every one of those the hard part was never the automation. It was proving the boundaries of the automation to someone who did not trust it.

What it does

What it does

Disputes are assessed the moment they arrive. Three agents read each one:

Intake classifies the complaint into a fixed vocabulary Investigation compares the customer's account against the bank's own transaction record Adjudicator proposes a ruling

None of them decides anything.

Every threshold, every deadline, and every naira of arithmetic is computed in deterministic Python. The tier that governs a dispute is a pure function of the amount:

$$ \text{tier}(a) = \begin{cases} 1 & a < 50{,}000 2 & 50{,}000 \le a \le 500{,}000 3 & a > 500{,}000 \end{cases} $$

Tier 1 resolves automatically. Tiers 2 and 3 require a human, regardless of how confident the model is.

The officer never processes disputes. They arrive at a queue where small claims have already resolved themselves and the cases the system refused to decide alone are waiting at the top. Their entire job is the escalations.

Every verdict carries a decision_rule field naming the specific rule that produced the outcome: tier_1_auto_resolve, tier_2_requires_human_approval, or tier_3_requires_human_approval. That field is computed in Python and rendered as its own column. The governance claim is not in the README. It is on the screen, on every row.

How we built it

Google ADK for the three agents, chained by plain sequential Python Runner calls Gemini 3.5 Flash on the global endpoint Firestore (native mode) as the system of record, with an audit subcollection per dispute FastAPI admin API, four endpoints, deployed on Cloud Run in us-central1 Vite / React / TypeScript / Tailwind console, served as an nginx container, also on Cloud Run

119 tests, the majority covering the deterministic core rather than the agents

A check_compliance.py script runs after every build stage. It seals critical files against known hashes, checks for hardcoded models and config, and refuses to go green on scope drift. Every change to a sealed file had to be authorised and re-sealed by hand.

Challenges we ran into

Agent Engine died silently. Deploying the agents to Agent Engine produced no traceback, no error, and no out-of-memory signal. The container was simply killed mid-way through the second conversational turn. The same code running in-process on Cloud Run succeeded 10 times out of 10. Diagnosing that meant building a deliberate probe rather than guessing, and the decision to abandon Agent Engine is logged with the evidence that produced it.

ADK's sequential agent construct failed quietly on two-turn tool calls. Replaced with explicit sequential Runner calls in plain Python. Less elegant, entirely debuggable.

The demo scene I had been describing did not exist. My project plan said a flagship case where the model proposed a refund at 0.98 confidence and a Python threshold overrode it. Late in the build I searched the entire repository for 0.98 and got zero results. No test, no fixture, no documentation. There was no comparison anywhere between what the model proposed and what a rule would have produced. The scene had been planned in conversation, sounded right, and been carried forward as done.

The real behaviour turned out to be stronger. On a ₦1,902,835 dispute the model proposed a refund at 0.95 confidence, and the system refused to execute it. Not because it doubted the model, but because confidence is not the deciding factor at that amount, and a person is. That is a truer claim and a better one for an enterprise governance category. Finding it a week before submission rather than after is the single most valuable thing that happened in this build.

The API had no filing date. Deadlines are computed from filed_at, and no dispute record contained one. The value had been passed in by hand from a throwaway script. Adding it meant editing a sealed protected file deliberately and re-sealing it.

Cloud Run's buildpack guessed wrong. It inferred a module named app; the application lives at src.api:app. The container exited before binding a port. There was also no requirements.txt anywhere in the repository. Every dependency had been installed straight into a virtualenv and never captured in a manifest.

A required schema field broke a working endpoint. Adding decision_rule as required meant three previously-written Firestore documents failed validation on the approval path, surfacing as a 500. Caught by tracing the code rather than by testing the happy path.

What we learned

Write the constraint down as an architectural rule, not a preference. "The LLM never does arithmetic or threshold decisions" stopped being a style choice the moment it was enforced by tests and a compliance script. It is the reason the governance story holds up under inspection.

Documentation drifts from code, and it drifts in the direction of what you wish were true. The PRD defines Tier 3 as above ₦500,000, fraud claims, or any guardrail trip. The code only ever checks the amount. Both gaps are now recorded as known gaps rather than quietly fixed in prose.

Investigate before asserting. Twice during this build, a confident hypothesis was overturned by reading the actual source. Both times the truth was more useful than the guess.

Known Gaps

I would rather state these than have a judge find them.

The guardrail seam at src/guardrails.py is called by Intake and Investigation before any customer text reaches a model, but it is currently a pass-through stub. Model Armor is not wired in.

Because compute_tier takes only an amount, a guardrail trip cannot yet escalate a dispute to Tier 3. Sample dispute D04 contains an embedded prompt injection instructing the system to credit ₦100,000 to an account. The Adjudicator ignored it entirely and ruled correctly on the actual complaint. It never receives raw customer text, only structured output from the two agents upstream. But the dispute auto-resolved at Tier 1 on amount alone, and nothing recorded that anything unusual was present.

Closing that gap means threading the screening result into tier assignment so that a trip forces human review. The seam is built for it. The wire is not yet attached.

What's next for Dispute Fleet

Wire the guardrail escalation path, then Model Armor behind it. Move decision_status derivation out of the pipeline and into the deterministic core where the rest of the threshold logic lives. And process disputes on arrival through a real queue, rather than one at a time. I want to test in with real world data and hopefully it is adopted into the banking sector in Nigeria

Built With

Share this project:

Updates

Submission history