ShiftLeft Society

PR comment with negotiation transcript
shiftleft-society architecture

What is ShiftLeft Society?

ShiftLeft Society is a multi-agent DevSecOps tribunal for GitHub pull requests. Two specialized AI agents, a Security Auditor and a Performance Analyst, review every PR independently. When they disagree about severity, they enter a structured negotiation with a confidence budget. The key design choice: the LLM only picks a categorical position (DEFEND, PARTIAL, or CONCEDE), while deterministic Python computes every consequence. That separation makes the whole verdict auditable and reproducible. A mediator produces the final call, and each agent's track record across every past PR adjusts how much budget it gets in future reviews.

Live at: https://shiftleft-society.duckdns.org
Live PR example: https://github.com/jmy744/shiftleft-society/pull/4

Track 3: Agent Society

The problem

Every engineering team uses AI code review now, but every current tool has the same weakness: a single AI voice making a single judgment with no cost to being wrong. Even the newer multi-agent tools (GitHub Copilot's parallel agents, for example) mostly divide labor. When two AI reviewers disagree about how serious an issue is, nothing resolves that disagreement in a principled, auditable way. It just becomes the developer's problem.

This matters more in 2026, because AI is writing more code than ever, so far more review is needed, and because automated decisions increasingly need a paper trail.

The engineering insight

The LLM proposes; deterministic code disposes.

Most "multi-agent negotiation" systems let the LLM decide everything, including the math. That means same input → different output every run. Unauditable.

ShiftLeft Society inverts that:

The LLM only picks a categorical position: DEFEND, PARTIAL, or CONCEDE
Deterministic Python computes the cost (DEFEND costs gap_tiers × 30, PARTIAL costs 15, CONCEDE costs 0), the revised severity, and the remaining budget
The LLM never touches a number

This gives 3 properties production systems need but agent demos rarely have: auditable, reproducible, defensible.

Cross-PR credibility memory

The negotiation isn't stateless. After every verdict, each agent's judgment is scored: did it defend correctly, or did it correctly defer to a peer flagging a more serious issue? That outcome is Bayesian-smoothed against a 5-negotiation neutral prior and capped at ±15 budget points, then persisted. The next time that agent negotiates on a completely different PR, it starts from 100 + track_record_adjustment. The system learns which of its own voices to trust, without any human re-weighting.

Visible in every PR comment as: "track record: 95% upheld over 42 past negotiations → budget +13."

Measurable improvement over single-agent baseline

40-case benchmark on Python-style vulnerabilities, same model and prompts for both:

System	Correct verdicts
Single-agent baseline	33 / 40 (82.5%)
Multi-agent tribunal	38 / 40 (95.0%)

+12.5 absolute points. The tribunal's biggest advantage is fewer false alarms: the negotiation cleared safe code (a hashed password, a verified JWT, enabled TLS) that the single agent wrongly flagged as dangerous.

Results committed to the repo: benchmark_results.json.

Architecture and tech stack

LangGraph orchestrates the agent society: parallel Round 1 (Send API), conditional Round 2 negotiation only when severity gap exists, then mediator
FastMCP (Model Context Protocol) exposes real security tools: scan_vulnerabilities, detect_secrets, check_yaml_pinning, analyze_complexity, with a local-regex fallback if the MCP server is unreachable
Qwen-Max via Alibaba Cloud DashScope (international endpoint dashscope-intl.aliyuncs.com) powers all agent reasoning
FastAPI gateway with SSE streaming, GitHub webhook receiver, SARIF and CycloneDX SBOM export
SQLite on a Docker-persisted volume for analyses, dialogue history, and the credibility table
Docker + Caddy for containerized deployment with automatic Let's Encrypt HTTPS
Alibaba Cloud ECS (Singapore region, Ubuntu 22.04) as the live deployment substrate

Engineering decisions

4-layer mediator fallback chain: structured output → raw parse → regex scrape → deterministic severity-tier inference. The mediator can never crash the run.
MCP fallback chain: every tool call degrades to local regex-based detection if the MCP server is unreachable.
Async correctness: blocking calls (GitHub diff fetch, SQLite reads) wrapped in asyncio.to_thread so they never stall the event loop. Background tasks held in a module-level set to prevent premature garbage collection. Webhook returns 200 fast, then processes and posts as a tracked background task within GitHub's timeout.
Standards-compliant output: SARIF 2.1.0 (GitHub Code Scanning compatible) and CycloneDX SBOM, so verdicts plug into existing enterprise tooling.
Persistent database: SQLite runs on a Docker volume so credibility data survives container restarts.

Developer workflow

A developer opens a PR as normal. Within about 10 seconds, a comment appears with the verdict, each agent's severity, an expandable negotiation transcript showing who argued what and what it cost, and copy-pasteable remediation code. No new tool to install, no dashboard to check, same place they already look. Cost is about two cents per PR at Qwen-Max pricing.

What I learned

3 lessons that shaped the design:

Non-determinism in LLM negotiation is a real problem. Letting the model self-report confidence numbers produces unreproducible verdicts. Constraining the LLM to categorical choices and computing consequences in code fixed this.
Error handling has to be visible, not just present. Building 4-layer fallbacks was easy; explaining them in the README so a judge can see the engineering was harder and more important.
Blocking calls in an async context silently break production systems. Every requests.get, sqlite3.execute, and unwrapped asyncio.create_task I found in early builds was a landmine. Wrapping them in asyncio.to_thread and holding background tasks in strong references is the difference between "works in demo" and "survives real webhook traffic."

limitations

Benchmarked on Python-style vulnerabilities. Other languages should work but are untested.
40-case benchmark is a solid start, not large-scale validation.
Single-instance SQLite deployment. Horizontal scale would need Postgres and a load balancer. The architecture supports this cleanly; it isn't deployed that way yet.
The credibility metric rewards categorical outcomes. A future refinement would ground it in actual merge outcomes and use proper calibration scoring (e.g., Brier score).

What I'd build next

1. Ground the credibility signal in real merge outcomes. Right now the credibility metric asks: "did this agent's severity match the final verdict?" That's a proxy. The real signal is what happened to the PR after the tribunal reviewed it. Did the developer merge it as-is, rewrite the flagged code, or ignore the review entirely? That's a fundamentally different feedback loop. The system stops guessing whether it was correct and starts learning from what humans actually did with its verdicts.

2. Multi-language through the MCP layer, not just prompting harder. The obvious "extend to JavaScript" answer is to prompt the LLM more aggressively. What's more interesting: add language-specific detectors (ESLint for JS, go vet for Go, cargo-audit for Rust) as MCP tools. The deterministic layer grows, the LLM guesses less, and the whole thing gets more auditable as it gets more general.

3. A third specialist agent to stress-test the negotiation mechanic. With only two agents, negotiation is 1-v-1. Adding a Maintainability voice tests whether confidence-budget generalizes cleanly to N-way disputes, or whether it needs redesigning at higher agent counts.

4. Team-level credibility, not just global agent-level. Different teams have different risk tolerances. Track credibility per-repo or per-team, and the system tunes itself to each team's actual priorities over time.