DevOps Swarm

6 AI agents analyze your PRs in parallel inside Slack — scanning for security vulnerabilities, deployment risks, SLO violations, and cost impact before you merge. Real-time. No context switching.

Comment

Inspiration

Every time our team merged a PR, we had to context-switch across four tabs — GitHub for the diff, Grafana for metrics, a spreadsheet for SLO tracking, and Slack to coordinate. The decision of whether to deploy lived across too many tools. We wanted the answer to come to us, not the other way around.

What it does

DevOps Swarm is a multi-agent AI system that lives entirely inside Slack. Type /swarm Namanjs/Docker-FAFO #3 and six specialized AI agents spin up in parallel — each analyzing a different dimension of your PR:

  • Security Scanner — detects hardcoded secrets, SQL injection, eval() usage
  • Deployment Safety — checks deployment windows, recent incident history, rollback feasibility
  • Incident Triage — correlates open incidents with the services your PR touches
  • SLO Compliance — flags if current error budgets are too tight for a risky deploy
  • Cost Intelligence — estimates infrastructure cost delta from IaC changes
  • Workflow Optimizer — audits CI/CD pipelines for security anti-patterns

Results are synthesized by an LLM into a single verdict — SAFE / NEEDS REVIEW / BLOCK — posted back to Slack as a rich Block Kit report, with a full Canvas document for audit trails.

How we built it

Built on the Slack Bolt SDK with Socket Mode for real-time event handling. The six agents run across three priority tiers using Promise.all, so Tier 1 (Security + Deployment) always completes before Tier 2 analysis begins.

All external tool calls go through an MCP (Model Context Protocol) subprocess — giving us process isolation, runtime tool discovery, and a clean authentication boundary. The MCP server exposes 10 tools across SQLite and the GitHub API.

We added deterministic guardrails on top of the LLM layer: if the Security Scanner flags a critical vulnerability, it programmatically caps the Deployment Safety score to ≤49 regardless of what the LLM outputs. This makes safety checks tamper-resistant.

The backend runs on Railway, the dashboard on Vercel (Next.js 14 with SSE-based real-time swarm visualization).

Challenges we ran into

  • LLM score inconsistency — models would describe a critical bug in prose but give an 80/100 safety score. We solved this with BaseAgent post-processing that scans the text for vulnerability keywords and clamps the score if they conflict.
  • Parallel agent failures — one slow GitHub API call could block the whole swarm. We wrapped every agent in fault isolation so failures are reported honestly without blocking other agents.
  • MCP subprocess lifecycle — managing stdin/stdout JSON-RPC with a circuit breaker that auto-restarts after 5 consecutive failures took significant iteration.

What we learned

Deterministic guardrails beat prompt engineering for safety-critical systems. LLMs are powerful summarizers but unreliable scorers — the real reliability comes from the code layer around them.

What's next

PagerDuty and Jira integrations (already stubbed), the Log Analysis and Capacity Planning agents (implemented but pending MCP tool registration), and a Slack Marketplace listing.

Built With

Share this project:

Updates