WASPADA

A six-agent AI society that debates every loan-risk score before a human decides.

Comment

WASPADA: a society of agents that argues about who to call

Inspiration

A collections team can only call a fraction of its delinquent borrowers each day. Picking which ones, and then defending that pick to a credit committee, is basically the whole job.

The trouble with a single risk score is that it arrives with no second opinion. No audit trail. No way to catch the expensive mistake, which is almost always the account that looks fine and quietly rolls into default.

So we flipped the question around. What if the model had to defend its score? Not to a dashboard, but to another agent whose entire job is to disagree with it. And when those two can't settle it, a human decides.

What it does

WASPADA runs a six-member society of specialized agents that produce, challenge, and defend a collections work-list.

Agent What it does Brain
Data Engineer Validates and profiles the freshly loaded loan book qwen3.6-flash
Data Analyst Builds features and the aggregates the debate later cites qwen3.7-plus
Actuary Scores every account, then defends that score when challenged sklearn + qwen3.7-plus
Risk Auditor Independently audits a slice and opens disputes where it disagrees qwen3.6-flash
Credit Arbiter Rules on any dispute the Actuary won't concede qwen3.7-max
Insight Assembles the work-list, released only through a human gate deterministic

The debate is load-bearing, not decoration. Disputes go through a bounded argument (auditor challenges, actuary rebuts, arbiter rules), and the ruling changes the actual decision, not just the transcript. Every claim has to cite evidence through a real MCP tool interface (portfolio_stats, lookup_account).

Governance is deliberately asymmetric, because the two errors don't cost the same. A ruling that raises risk applies automatically. A ruling that would cancel a collector call has to clear a human first.

Sitting above all of it is a parameter matrix the analyst controls. You set the risk-band to action grid, the dispute admissibility gap, the arbiter's escalation threshold, and the audit budget before a run, and every decision comes back stamped with the governing policy_id. The human isn't just approving one call. They're setting the rules the whole society plays by.

The portfolio view surfaces cohort alerts and expected loss per account and across the book:

$$EL = PD \times LGD \times EAD$$

where $PD$ is the calibrated default probability, $LGD = 0.45$, and $EAD$ is outstanding principal.

How we built it

We started with a deterministic spine, not an agent free-for-all. A plain orchestrator runs a fixed step order over a frozen data contract (RawLoans to FeatureFrame to ScoredAccounts to DashboardPayload). Agents reason inside steps, but the pipeline itself stays reproducible. The contracts allow additive supersets, which is why we shipped six features without a single breaking migration.

We used classical ML where classical ML still wins. The PD model is logistic regression with a vintage split and an explicit leakage guard. AUC came out at 0.967, and post-hoc isotonic calibration pulled Brier from 0.087 down to 0.028, so p_default is a real probability rather than a vibe. The model is version tracked and drift monitored with per-feature PSI, all surfaced as a model card. We score 100% of the book classically and only spend LLM budget on the handful of accounts that are actually contested.

Qwen models are tiered by how much thinking the job needs. Flash for triage, plus for analysis and defense, max for the arbiter's rulings, all through native function-calling loops.

Alibaba Cloud runs the whole thing. FastAPI on Function Compute in a custom container pulled from Container Registry. The real Lending Club book (50,000 loans) lives in OSS across raw, staging, and mart medallion buckets and gets read into an in-process DuckDB. RDS MySQL backs auth, SLS captures every run's audit trail, and it's all declared in OpenTofu.

Then the same society picked up a second job. We pointed it at Origination (approve, refer, reject) with an application-time model, and the debate, the gate, and the dashboard were reused as-is. A whole second product lane landed with zero infrastructure change.

546 tests, all green.

Challenges we ran into

We shipped fake data without realizing it. Our demo fixture looked completely real. It wasn't. A config check looked for OSS_BUCKET while the environment actually set OSS_RAW_BUCKET, so every single generation run quietly fell through to a synthetic sample instead of the real book. We only caught it because Indonesian province names turned up in a US loan portfolio. One word of config, and we'd been demoing invented data.

Real data bites back, and an agent caught it before we did. Once we ran against the actual book, the top of the work-list filled up with accounts sitting at p_default = 1.00. The raw Lending Club data includes loans that were already charged off, which the model correctly scores as certain default but which are useless as collections targets. The genuinely great part is that the Risk Auditor found this on its own, arguing that "the outstanding_ratio of 0.00 indicates the principal balance is fully extinguished, rendering the forward-looking probability of default negligible." An agent caught a data artifact we had walked straight past.

You can't make a live stream feel human. We wanted people to actually watch the society think. But the SSE stream can't guarantee readable pacing. The round-one challenges get produced by a parallel 8-worker audit and land in one burst, and serverless proxies can collapse whatever's left. So we inverted it and treated the stream as an ordered data source, then drove the reveal off a client-side clock instead. Now each turn appears one at a time with a "is thinking" beat, and the analyst gets play, pause, step, and replay. Automation they're in control of.

Serverless networking nearly ate the last night. Function Compute has to sit inside a VPC to reach RDS on the private network, but a VPC-bound function has no internet route, so every live Qwen call died with "brain unreachable." That kicked off a long descent through NAT gateways, elastic IPs, SNAT entries, route tables, and a few Forbidden.RAM permission walls. The lesson we'd hand to anyone: your demo's critical path should never depend on your least reliable dependency.

What we're proud of

Disagreement is a first-class state here, not an error we swallow quietly. When the arbiter rules, the work-list actually changes, so the debate is decision-making rather than theater. The model is calibrated, version tracked, and drift monitored, which is the bar a real risk committee would hold us to. And the society generalized to an entirely different lending decision without us touching the architecture.

What we learned

Multi-agent systems need a deterministic spine and frozen contracts, otherwise they're unreproducible and miserable to debug.

Asymmetric decisions deserve asymmetric governance. Auto-applying a risk increase while gating a risk decrease is one small rule that encodes a very real operational cost.

Real data is a collaborator. Synthetic data would never have taught us about charged-off contamination, and it would never have given the auditor something true to catch.

Pace is a design decision. People trust what they can watch happen. Slowing the debate down made it more convincing, not less.

What's next

A proper pending-approval queue, so the human gate is genuinely interactive instead of resolving inline. Scheduled ingestion on a timer rather than on demand. And reject-inference for the origination lane, since today's label is funded-then-defaulted, which is honest but incomplete.

Built With

Share this project:

Updates