The Council

A council of specialized AI agents that finds more bugs than any single reviewer — and debates the tradeoffs.

Comment

Inspiration

Most AI code review tools compress everything into one homogenized pass — security concerns get diluted by style nitpicks, and a single model's blind spots become the whole review's blind spots. We wanted to see what happens when you don't ask one AI to be everything at once. What if four agents, each stubbornly specialized, had to actually defend their recommendations against each other? Track 3's explicit ask — measurable efficiency gains and real conflict resolution, not just task division — pushed us to build something where disagreement isn't a bug, it's the point.

What it does

The Council runs four Qwen-powered agents — Security, Performance, Readability, and Architect — against submitted code (JavaScript, TypeScript, Python, or C) in parallel. Each agent is restricted to its own domain by both prompt design and a programmatic guard, since even well-prompted models will report "obvious" issues outside their lane. When two agents' recommendations genuinely conflict — say, Security wants a synchronous check that Performance can prove costs real latency on a documented hot path — they get a debate round to argue it out with concrete numbers, not vague concern. A Moderator agent then synthesizes everything into a prioritized verdict with explicit conflict resolutions. A solo baseline reviewer runs in parallel (zero added latency) so the UI shows exactly how many more issues the council caught over a single generic pass — often 3-4x more.

The Council is also available as a VS Code extension — right-click any selection, run 'The Council: Review selection', and the same four-agent debate runs directly against your local backend, with results shown in an editor-side panel.

How we built it

  • Backend: Node.js + Express, streaming every step of the pipeline to the frontend via Server-Sent Events so the debate feels live rather than a 25-second black box. Agent orchestration hits Qwen models through Alibaba Cloud Model Studio's OpenAI-compatible endpoint.
  • Frontend: React + TypeScript + shadcn/ui, with a resizable split-panel debate view that adapts to mobile. The conflict detection layer went through several iterations. Our first version flagged 8 "conflicts" on a single review — almost all false positives from keyword overlap in verbose reasoning text. We tightened it to match only on the short issue field and require genuine topic overlap between findings, which took us to zero conflicts — technically correct (agents were just agreeing), but useless for demonstrating the "resolve disagreements" requirement. We had to deliberately construct a snippet with an unambiguous, quantified tradeoff (a synchronous fraud-check API call costing 80-150ms on a documented 50ms-SLA checkout path) before the agents had anything real to argue about.

Challenges we ran into

The Alibaba Cloud KYC/payment-method verification requirement turned out to be a widespread blocker — documented across dozens of hackathon Discord participants worldwide hitting Forbidden.RiskControl errors and rejected verifications, often with no card at all. We had to route around it by using Model Studio's API-level access (which worked without full account verification) rather than deploying a full ECS instance, and reached out to organizers directly given the FAQ's contradictory wording on whether ECS deployment was mandatory or a bonus. On the technical side, getting agents to genuinely disagree — rather than either all agreeing or one agent silently reporting another's domain — required both careful prompt boundaries ("SecurityAgent owns injection/secrets exclusively") and a backup keyword-based filter, since prompt instructions alone weren't reliable when an issue was "too obvious" to skip.

Accomplishments that we're proud of

A fully working, streamed multi-agent debate pipeline that produces genuinely reasoned conflict resolutions — not templated diplomacy. On our benchmark snippet, the Moderator correctly weighed a real security-vs-performance tradeoff with a specific, defensible number ("parameterization adds ~0.3ms, not the ~8ms Performance estimated") rather than just picking the higher-severity label. That's the kind of judgment a single-pass reviewer doesn't produce. We also shipped a working VS Code extension on top of the same backend — proving the council's value isn't locked to one interface. Select code, run one command, get the full debate without leaving your editor.

What we learned

Getting AI agents to have a real disagreement is harder than getting them to agree — and that's a good sign, not a flaw. Most competent fixes genuinely don't have costly tradeoffs, so demonstrating conflict resolution meaningfully requires either scale (many real-world code samples) or intentionally engineered scenarios. We also learned that "the model was told not to do X" is never sufficient on its own for anything safety- or scope-critical; programmatic guardrails matter even with well-written prompts.

What's next for The Council

Direct GitHub PR integration so reviews trigger on real pull requests instead of pasted snippets. Persistent memory across review sessions so the council learns a codebase's conventions over time. Broader benchmarking across a larger corpus of real-world vulnerable code to get a statistically solid "vs solo" number instead of a single demo case. We already shipped a first VS Code extension (see /vscode-extension) that runs reviews on-demand from the command palette. Next: inline diagnostics using VS Code's native squiggly-underline API instead of a separate panel, live streaming of the debate as it happens (matching the web app's real-time feel), and eventually publishing to the VS Code Marketplace.

Built With

Share this project:

Updates