RedCouncil

Inspiration

High-stakes business decisions — pricing changes, layoffs, market entry, vendor switches — routinely get evaluated through a single point of view, even when an AI is doing the reviewing. Real boardrooms don't work that way: they put specialists with genuinely conflicting mandates in the same room and let them argue before anything gets committed. We wanted an AI system that worked the same way — not one model producing a confident-sounding summary, but five agents forced to defend, and attack, each other's positions.

What it does

RedCouncil runs five specialized agents — Growth, Risk, Legal, TechDebt, and Customer — against a business decision in parallel. Each forms an independent position from its own mandate before seeing any other agent's view. Then they cross-examine each other across structured debate rounds, a conflict detector flags genuine disagreement between them, and a Synthesizer produces a single severity-scored vulnerability report — with every surfaced conflict laid out, not buried in a single-paragraph summary.

How we built it

Five agent personas and a Synthesizer orchestrated as a LangGraph state machine, running parallel fan-out for independent analysis and structured fan-in for cross-examination. All six roles run on Qwen Cloud (Qwen-Max). FastAPI backend, PostgreSQL for persistent case storage, Clerk for auth, deployed end-to-end on Alibaba Cloud ECS. Conflict detection uses a deterministic stance-polarity check rather than an LLM judge, and the Synthesizer runs a bounded, coverage-checked retry loop so no specialist's findings can silently disappear from the final report.

Challenges we ran into

Our conflict detector originally filtered candidate disagreements through a lexical-overlap gate — but agents with genuinely different mandates use genuinely different vocabulary by design (Growth talks about savings, Customer talks about churn), so real disagreements between them were getting filtered out before they ever reached the polarity check. We fixed it by trusting each agent's own explicit rebuttal signal from cross-examination instead of a token-overlap heuristic.

We also found that our Synthesizer could silently drop an entire specialist's perspective from the final report if a coverage-check retry failed to recover it — invisible unless you were specifically diffing agent input against report output. We only caught it by building a deterministic single-agent baseline comparison and noticing our own system was, in a few scenarios, covering less ground than one plain model call. That comparison became core to how we validate the system now, not just a one-off test.

Working with metered Qwen Cloud credits also meant every fix had to be validated deterministically before we'd spend a review re-running the full comparison — cheap, local, no-LLM scoring wherever possible.

Accomplishments that we're proud of

• A deterministic, measured comparison against a single-agent baseline on the same model — not just an assertion that "multi-agent is better." RedCouncil surfaced 25 cross-agent conflicts across 5 real business scenarios that a single model call is structurally incapable of detecting, and caught 62% more high-severity risks on average.

• A fully deployed, working system on Alibaba Cloud — not a local-only demo.

• A synthesis pipeline that's auditable: every red flag and conflict in the final report can be traced back to which agents raised it and why.

What we learned

Adversarial multi-agent architectures need to be validated the same way any hypothesis is — with a controlled baseline holding the model constant — not just judged on whether the output feels more thorough. We also learned that the real failure modes in a system like this are rarely about reasoning quality; they're about plumbing. The same result computed correctly in one code path and silently dropped in another cost us more debugging time than any prompt-engineering problem did.

What's next for RedCouncil

Expanding the scenario library across more industries and company sizes, adding observability into how the Synthesizer weighs conflicting agent findings so users can see why a report reached its conclusion, and exploring human-in-the-loop checkpoints where a real reviewer can inject a rebuttal directly into an active debate round.

Built With

• alibabacloud
• clerk
• fastapi
• langgraph
• postgresql
• pydantic
• python
• qwen
• qwen-max
• react