SentrySwarm

Your AI SRE team that investigates, diagnoses, and resolves production incidents autonomously.

Comment

SentrySwarm — Autonomous AI SRE

Inspiration

Production incidents are rarely caused by one obvious problem.

When an application goes down or becomes slow, engineers often have to investigate logs, code changes, Kubernetes, cloud infrastructure, deployments, security events, and system metrics across multiple tools before they can identify the root cause.

This inspired us to ask:

What if a production system had its own AI SRE team that could investigate an incident instead of waiting for a human engineer to do everything manually?

We wanted to explore whether a swarm of specialized AI agents could work together like a real engineering team.

What We Built

SentrySwarm is an autonomous AI SRE system where specialized agents collaboratively investigate, diagnose, and remediate production incidents.

Each agent has a focused responsibility:

  • 🔍 Log Agent — analyzes application and system logs
  • ☸️ Kubernetes Agent — investigates pods, deployments, and resource health
  • ☁️ Cloud Agent — checks cloud infrastructure and service health
  • 💻 Code Agent — analyzes recent code changes
  • 🔐 Security Agent — investigates potential security-related causes
  • 🚀 Deployment Agent — correlates incidents with recent releases
  • 🧠 Investigation Agent — combines evidence and determines the most likely root cause
  • 🛠️ Remediation Agent — determines and executes the safest recovery action
  • ✅ Verification Agent — confirms that the system has recovered

The key idea is that these agents do not work in isolation.

They investigate different parts of the system, share their findings, challenge hypotheses, and collaborate to reach a reliable diagnosis.

How It Works

                    🚨 Production Incident
                             |
                             v
                    ┌─────────────────┐
                    │  Swarm Manager  │
                    └────────┬────────┘
                             |
          ┌──────────────────┼──────────────────┐
          v                  v                  v
     🔍 Log Agent       ☸️ K8s Agent       ☁️ Cloud Agent
          |                  |                  |
          └──────────────────┼──────────────────┘
                             |
                  ┌──────────┴──────────┐
                  v                     v
             💻 Code Agent        🔐 Security Agent
                  |                     |
                  └──────────┬──────────┘
                             v
                   🧠 Investigation Agent
                             |
                             v
                    Root Cause + Evidence
                             |
                             v
                    🛠️ Remediation Agent
                             |
                             v
                       Fix / Rollback
                             |
                             v
                    ✅ Verification Agent
                             |
                             v
                    Incident Resolved

For example, if API latency suddenly increases, the swarm can investigate:

  1. Recent deployments
  2. Application error logs
  3. Kubernetes health
  4. Cloud infrastructure
  5. Recent code changes
  6. Security events
  7. Resource and service behavior

Instead of asking a single AI model to guess the problem, specialized agents investigate different dimensions and provide evidence to the swarm.

Example Incident

Imagine a new application version is deployed.

A few minutes later:

🚨 API latency increases dramatically.

The swarm starts investigating.

Log Agent:

Increased database connection errors detected.

Kubernetes Agent:

Pods are healthy, but requests are timing out.

Deployment Agent:

Incident began shortly after the latest deployment.

Code Agent:

Database connection handling changed in the latest commit.

Cloud Agent:

CPU and memory are within normal limits.

The investigation agent correlates the evidence and identifies:

Root cause: the latest application release is exhausting database connections.

The remediation agent recommends rolling back the deployment.

After rollback, the verification agent checks the system and confirms that latency has returned to normal.

The result is an evidence-driven incident response rather than a single AI-generated guess.

What Makes SentrySwarm Different?

Traditional incident response often looks like:

Alert
  ↓
Human engineer
  ↓
Check dashboards
  ↓
Check logs
  ↓
Check deployment
  ↓
Check infrastructure
  ↓
Find root cause
  ↓
Fix
  ↓
Verify

SentrySwarm transforms this into:

Alert
  ↓
AI Agent Swarm
  ↓
Parallel Investigation
  ↓
Evidence Sharing
  ↓
Collaborative Root Cause Analysis
  ↓
Safe Remediation
  ↓
Automated Verification

The swarm architecture is important because production incidents require knowledge from multiple domains.

A single general-purpose agent may not reliably investigate every layer of a modern application. Specialized agents can focus on their domain while the swarm combines their evidence.

What We Learned

Building SentrySwarm taught us that multi-agent systems are most useful when different agents have clearly defined responsibilities and need to collaborate to solve a problem.

We also learned that giving an AI agent access to tools is not enough.

Agents need:

  • Clear roles
  • Structured outputs
  • Relevant tools
  • Evidence-based decisions
  • Coordination
  • Controlled permissions
  • Verification before and after actions

Most importantly, we learned to think of AI agents less like chatbots and more like members of an engineering team.

Challenges

1. Safe Autonomous Actions

Giving an AI system the ability to modify production infrastructure introduces risk.

A wrong diagnosis could make an incident worse.

We therefore designed SentrySwarm around evidence, confidence, controlled actions, and verification rather than allowing an agent to blindly execute commands.

2. Agent Coordination

Multiple agents investigating the same problem can produce duplicated or conflicting conclusions.

We addressed this by giving every agent a specialized responsibility and using a coordination layer to aggregate their findings.

3. Root Cause vs. Correlation

Finding that two events happened at the same time does not necessarily mean one caused the other.

SentrySwarm therefore considers multiple signals—including deployment history, code changes, infrastructure health, and logs—before reaching a conclusion.

4. Building a Realistic Demonstration

We wanted to demonstrate the entire lifecycle rather than simply show an AI-generated explanation.

Our target workflow is:

Incident
   ↓
Detection
   ↓
Investigation
   ↓
Evidence Correlation
   ↓
Root Cause
   ↓
Remediation
   ↓
Verification
   ↓
Resolution

Vision

SentrySwarm is our first step toward autonomous reliability engineering.

Our vision is simple:

Every production system should have an AI SRE team capable of investigating incidents, collaborating on evidence, taking safe action, and verifying recovery.

In the future, SentrySwarm could integrate with cloud providers, Kubernetes clusters, CI/CD systems, observability platforms, databases, and security systems to provide autonomous reliability engineering across the entire software lifecycle.

Final Thought

We are not building another chatbot that tells engineers what might be wrong.

We are building a team of AI agents that investigates the problem together.

SentrySwarm: Give your production system an AI SRE team.

Built With

Share this project:

Updates