Synod

• 

  synod cli chat

Inspiration

Track 3 asked for agents that collaborate through "task division, dialogue, and negotiation." Most multi-agent demos I'd seen took that literally — several agents, all firing in parallel, glued together with a summary at the end. That's not collaboration, that's five monologues happening at the same time.

I wanted agents with an actual reason to depend on each other, and I wanted to measure whether that dependency made the output better — not just assume it did because it sounded sophisticated.

What it does

Synod is a multi-agent code review system. Four specialized agents run in a deterministic pipeline instead of a free-for-all debate:

• Cartographer maps the code's structure first — modules, dependencies, entry points — before any judgment happens.
• Inspector and Sentinel then analyze in parallel, using that map as shared context. Inspector checks code quality (complexity, anti- patterns, maintainability). Sentinel hunts CWE-mapped security vulnerabilities, backed by a Semgrep static-analysis pre-filter that gives it a deterministic floor to work from.
• Arbiter deduplicates their findings, validates every cited line number against the real source file (dropping anything hallucinated), and escalates severity when two agents independently flag the same issue.
• Smith, optional, generates concrete fixes for high-severity findings — and Sentinel re-reviews each fix before it's accepted, in a bounded retry loop.

It ships as a REST API, a terminal CLI (review a file, chat, or scan a whole directory), and a GitHub webhook that posts findings directly as PR comments.

How I built it

I started with something much bigger — six agents, four rounds of debate, episodic and semantic memory with a vector database. It looked impressive on a whiteboard. Then I benchmarked it.

The six-agent debate version found more things, but precision had collapsed to near zero — it was drowning three real vulnerabilities in a hundred hallucinated ones and calling it thoroughness. That result forced a full rebuild: I cut it down to four agents with clear, non-overlapping roles, replaced the debate rounds with structural dependency (Cartographer's output actually feeding the other agents, not just being displayed alongside them), and added Arbiter's evidence-validation step so a finding can't survive if its line number doesn't exist in the actual file.

Every architectural decision after that got tested, not assumed. When I added the Semgrep pre-filter, my first version made things worse on one sample — precision dropped from 100% to 67% because overlapping Semgrep rules were injecting duplicate, unvalidated findings straight into the output. I diagnosed the root cause, added deduplication by CWE-plus-line clustering, and required Sentinel to validate every Semgrep candidate against the real code before accepting it. Re-benchmarked: recall went to 100% with precision recovering to 83%, consistently, run after run.

Challenges I ran into

Scale isn't signal. More agents and more debate rounds produced more noise, not more truth. The fix was cutting complexity, not adding it.

Model compatibility is not guaranteed. Not every Qwen model reliably returns strict JSON for the agents' structured prompts — some general- purpose models returned None for Inspector and Sentinel entirely. I ended up splitting configuration into a main/chat model and a dedicated JSON-agent model, so the system stays usable even when the "smartest" available model isn't the most reliable one for structured output.

Quota is a real constraint, not an afterthought. Running a proper benchmark — three runs per sample, multiple conditions — burns tokens fast. I had to build the benchmark script to save results incrementally, estimate cost before running, and support re-running just the failed sample instead of the whole suite, after losing partial runs to exhausted free-tier quota more than once.

Sycophancy in agent debate is real. My original four-round debate design assumed more discussion would converge on better answers. It didn't — agents tended to agree with whatever the majority said rather than with what was actually true. Removing the debate and replacing it with one-way structural dependency and a hard evidence check produced better results with less cost.

Accomplishments I'm proud of

Publishing a formal benchmark with actual TP/FP/FN definitions, three runs per sample, mean and standard deviation — and being honest in the README about what doesn't work yet (CSRF detection is still weak) instead of only showcasing the wins. Also proud that the single-agent-vs-council comparison shows something concrete: single-agent recall varied between 0% and 75% across runs on the same file, while the Semgrep-backed council held 100% recall every single time. That's the actual argument for why this architecture exists.

What I learned

That an agent society is worth building when agents genuinely depend on each other's output — not when there are simply more of them. That deterministic tools and probabilistic models solve different problems, and the right answer is usually both, each doing what it's actually good at. And that a benchmark you're willing to publish, warts included, is worth more than a demo that only shows the best run.

What's next for Synod

Weighted voting based on each agent's historical accuracy, expanding past Python to JS/TS, Go, and Rust, and a GitHub Action for fully automated CI review — all deferred deliberately, so the current version stays small enough to actually trust.

Built With

• alibaba-cloud
• asyncio
• cli
• dashscope
• docker
• ecs
• fastapi
• github-api
• github-webhooks
• hmac
• httpx
• json-schema
• openai-sdk
• pydantic
• pytest
• python
• python-dotenv
• qwen
• rest-api
• rich
• semgrep
• typer
• uvicorn
• yaml