SRE Agent

• 

  SRE Agent

Inspiration

On-call at 3 AM, an alert fires. The engineer spends 35 minutes manually checking New Relic, PagerDuty, GitLab, Jira, and Confluence — context-switching across 5 tools just to answer one question: what broke and why?

We wanted to eliminate that 35 minutes entirely.

What it does

SRE Agent is a Slack bot that automatically investigates production incidents the moment an alert fires. It joins the incident channel, runs a full root-cause investigation across 9+ tools in parallel, and posts a structured report — including customer impact, likely cause, and recommended fix — in under 60 seconds.

How we built it

• Skill-based routing: the alert is matched against a corpus of 500+ historical incidents to pick the most relevant investigation strategy (deployment regression, infra failure, third-party outage, etc.) — at zero API cost.
• ReAct agent loop: Claude Sonnet drives tool calls iteratively — reasoning about what to check next based on each observation.
• Parallel tool execution: New Relic golden signals, deployments, NRQL queries, PagerDuty incidents, GitLab pipelines, Confluence PIR archive, and Jira RCA tickets are all fetched in parallel.
• Customer impact meter: real-time scoring of how many orgs are affected, surfaced in every report.

Challenges we faced

• Getting the agent to know when to stop — avoiding over-investigation while still being thorough.
• Keeping latency under 60 seconds with 9+ tool calls, which required careful parallelization.
• Parsing freeform Incom alert messages reliably across dozens of alert formats.

What we learned

Skill-based routing before the LLM loop is a huge force multiplier — injecting a focused hypothesis into the system prompt makes the agent dramatically more accurate and efficient than a blank-slate investigation.

ROI

Built With

• ai
• cloud
• incident
• sre