Adversarial-Integrity-Audit-System

Five AI agents scan a target, then debate every finding like a courtroom - red argues worst-case, blue argues mitigation, and the room votes on the truth.

Comment

Inspiration

The spark for this came from a question that doesn't get asked enough in security tooling: what happens when the scanner is the thing that's wrong?

Every audit pipeline quietly assumes its tools are honest. But a compromised dependency feed and a scanner that's just quietly misconfigured produce the exact same downstream signal - a real vulnerability that never shows up in the report. We realized we didn't actually need to solve the harder problem of detecting intent. We only needed a system that didn't care whether the corruption was malicious or accidental, and that could catch either one before it reached a human. That reframing - stop asking "who's lying," start asking "did the damage get through" - became the whole design principle for the project, and it's the reason this had to be a society of agents arguing with each other rather than one clever model trying to be right alone.

Track 3, Agent Society, was a natural fit: we wanted to build something where the interesting behavior only exists because there's more than one agent in the room.

What it does

Five scanner agents - network, web app, code, config, and dependency - each wrap a real security tool (Nmap, OWASP ZAP, Semgrep + Bandit, custom Docker/K8s/IAM checks, Trivy) and independently score every finding they produce, \( s_{raw} \in [0.0, 1.0] \), before any of them talk to each other. Those scores get weighted by each agent's own historical accuracy \( w_i \), loaded from past sessions:

$$s_{weighted} = s_{raw} \times w_i$$

so a scanner with a track record of over-flagging noise (say \( w_i = 0.60 \)) simply carries less influence in the room than one with a track record of being right (say \( w_i = 0.95 \)), without anyone hand-tuning that trust.

A configurable error injection layer sits in the middle of the pipeline and deliberately corrupts a fraction of findings - malformed data, a real vulnerability quietly rescored as harmless, a finding wearing another agent's name tag, an inverted severity label, or a mixed batch of clean and poisoned findings. This is the benchmark: the agent society wins a round if every high-severity injected corruption still surfaces correctly in the final report.

Before anything reaches an LLM or the message bus, a tokenization layer strips and replaces every sensitive value (IPs, hostnames, credentials, connection strings) with a session-scoped token, backed by an encrypted vault. Only the final PDF report is ever allowed to reverse that.

Surviving findings go through a structured debate: a Red position argues maximum severity and builds an exploit chain, a Blue position argues compensating controls, and any agent whose score diverges from the group mean \( \bar{s} \) by more than a threshold,

$$|s_i - \bar{s}| > 0.25$$

is forced into a formal cross-examination — it must challenge or concede, on the record, with its reasoning logged. The final majority vote \( v \) then resolves the finding into a CVSS band:

$$ \text{severity}(v) = \begin{cases} \text{CRITICAL} & v \geq 0.90 \ \text{HIGH} & 0.70 \leq v < 0.90 \ \text{MEDIUM} & 0.40 \leq v < 0.70 \ \text{LOW} & 0.10 \leq v < 0.40 \ \text{INFO} & v < 0.10 \end{cases} $$

No agent has structural authority in that vote - severity is earned by the room, never hardcoded.

After each completed session, every agent's trust weight recalculates from its real track record - including a term that specifically rewards an agent whose lone dissenting vote was later vindicated by a confirmed finding. A live dashboard streams the whole debate - agent status, red/blue arguments, vote resolution - as it happens.

How we built it

We split the work three ways around a single hard dependency: nothing else can start until the Pydantic schemas exist, because every message between every component has to validate against them. Person 1 owned that foundation - schemas, Postgres/Redis clients, the tokenization vault, the session state machine, and the recalibration formula - and shipped it day one specifically to unblock the other two. Person 2 built everything that produces findings: the five scanner agents, the LangGraph orchestrator, the error injection layer, the cloud connectors, and the Qwen client. Person 3 built everything that makes the findings visible: the debate engine, the FastAPI routes and WebSocket stream, and the React dashboard - starting on prompt templates from day one, since that piece had no upstream dependency.

The system runs as a single LangGraph StateGraph: initialize → preflight → parallel scan → inject → tokenize → screen → debate → vote → ledger → recalibrate → report. Every session moves through an explicit state machine (INITIALIZING → SCANNING → SCREENING → DEBATE → FINALIZING → COMPLETE / PARTIAL / ABORTED), enforced both in code and as a Postgres check constraint, so a session can never contribute to recalibration unless it actually finished cleanly.

We matched three tiers of Qwen to three different jobs: Qwen3.5-Flash for the five agents' fast, structured scoring; Qwen3.5-Plus for the sub-orchestrator, which needs a long context window to summarize the last 10–15 session logs before a new debate starts; and Qwen3-Max for the debate rounds themselves, where the adversarial reasoning actually needs to hold up. Everything runs on Alibaba Cloud - ApsaraDB for PostgreSQL as the findings ledger and weight store, ApsaraDB for Redis as the inter-agent bus, ACR for images, KMS for secrets, and a RAM service account per container so no two components ever share credentials.

Challenges we ran into

Keeping the debate honest. It was tempting to give the sub-orchestrator some kind of tie-breaking authority "just in case." We deliberately didn't - no agent has structural authority over the vote, which meant we had to actually trust the weighting and cross-examination logic to resolve disagreements instead of falling back on a hardcoded judge.

Sensitive data near an LLM. Real scan output is full of real IPs, hostnames, and credentials, and none of it can touch a model or the Redis bus in plaintext. Getting the tokenization layer deterministic within a session (same value, same token) but distinct across sessions - while still supporting a stable, session-independent token for cross-session recalibration joins - took more design passes than we expected before it stopped leaking edge cases.

Rewarding dissent instead of punishing it. Early recalibration formulas naturally converged toward "the agent that agrees with the room most often wins." We had to deliberately build in an overrule term so that an agent whose minority vote was later vindicated gains weight rather than losing it for being an outlier:

$$ \text{accuracy_rate} = \frac{\text{correct verdicts}}{\text{total verdicts}} \qquad \text{overrule_rate} = \frac{\text{times overruled and correct}}{\text{total times overruled}} $$

$$w_{new} = \text{clamp}\big(0.7 \cdot \text{accuracy_rate} + 0.3 \cdot \text{overrule_rate},\ 0.1,\ 1.0\big)$$

Without that second term, the system would slowly train itself to stop listening to its most useful contrarian.

Coordinating three people around one blocking file. The whole first week hinged on schemas.py landing correctly on day one - a late or wrong schema would have cascaded into everyone else's work. We treated that file almost like a contract negotiated up front rather than something to iterate on casually.

Budgeting for the debate. Qwen3-Max is the expensive, high-quality tier, and debate rounds are the most token-hungry part of the whole pipeline. We had to be disciplined about temperature (0.2, for consistent security reasoning) and prompt size to keep a full session affordable across the hackathon's cloud credit.

Accomplishments that we're proud of

Getting the "malicious scanner and misconfigured scanner look identical" framing to actually hold up end-to-end, not just as a nice sentence in the README the pipeline genuinely doesn't need to know which one it's looking at. We're also proud of the vindication mechanic: a dissenting agent's argument doesn't just get logged and forgotten, it can concretely earn that agent more influence in the next session. And we're proud that the credential isolation isn't cosmetic only one identity in the entire system can write to the trust-weight table, which is the actual thing stopping a compromised agent from rigging its own future votes.

What we learned

That reframing a hard problem ("is this attacker or accident?") into an easier, answerable one ("did the damage get through?") can unlock an entire architecture - the whole debate-and-recalibration design only exists because we stopped trying to detect intent. We also learned a lot about matching model tiers to the actual shape of the task instead of defaulting to the biggest model everywhere: Flash for parallel structured scoring, Plus for long-context summarization, Max only where the reasoning quality is what's actually being judged. And we came away with a much sharper appreciation for how much a multi-agent system's fairness depends on its incentive structure - a debate protocol is only as good as the weighting formula deciding whose voice gets heard next time.

What's next for Agent Society

Running enough real sessions to see the recalibration weights genuinely diverge from their defaults and beat the single-agent baseline by a clear margin - five sessions is our working minimum, but we want more data before we trust the curve. We'd also like to open the anomaly-screening thresholds up to tuning per target, add a sixth or seventh agent (secrets-in-transit, cloud storage misconfig) while keeping the odd-number rule for clean majority votes, and extend the vindication workflow so a confirmed post-audit breach can automatically trigger recalibration instead of waiting for the next scheduled session.

Built With

  • adversarial-ai
  • agent-society
  • agentic-workflows
  • ai-debate
  • aidebate
  • alibaba-cloud
  • anomaly-detection
  • cyber
  • cybersecurity
  • data-tokenization
  • hackathon
  • multi-agent-systems
  • network
  • qwen
  • qwen3-max
  • security-audit
  • self-improving-systems
  • vulnerability-scanning
Share this project:

Updates