Enterprise incident response tools assume mature SRE organizations with large budgets. For smaller IT and DevOps teams, incident response is a chaotic scramble of noisy channels, fragmented logs, and zero documentation. We wanted to democratize enterprise-grade incident response by building an automated, AI-driven Incident Commander that resides entirely within Slack.
We structured the application as a deterministic orchestrator that routes Slack Socket Mode events to four specialized, asynchronous AI agents running on Llama 3.3 and Llama 3.1 via Groq (with Gemini 2.0 as a robust fallback):
TriageAgent: Classifies severity rubrics and auto-allocates slack resources. TimelineAgent: Translates engineer updates into structured chronologies. InvestigationAgent: Gathers commit history and Slack conversations. DocumentationAgent: Synthesizes blameless postmortems. By using the Model Context Protocol (MCP), the app spawns a stdio-based GitHub server to fetch code logs. To optimize performance, we parallelized our evidence collection. Under sequential execution, the triage latency $T_{\text{seq}}$ is:
$$T_{\text{seq}} = T_{\text{RTS}} + T_{\text{commits}} + T_{\text{PRs}}$$
With our parallelized async loop, the triage latency $T_{\text{parallel}}$ is bounded only by the slowest request:
$$T_{\text{parallel}} = \max(T_{\text{RTS}}, T_{\text{commits}}, T_{\text{PRs}})$$
This yields a significant speed improvement:
$$\Delta T = \sum_{i=1}^{k} T_i - \max_{1 \le i \le k}(T_i)$$ Some Challenges we faced: Async SDK Mismatches: Under Socket Mode, the Slack Bolt framework dynamically wraps incoming client calls in an AsyncWebClient. Our initial SDK wrapper attempted to offload blocking tasks to a thread pool, returning a coroutine object instead of executing it. We resolved this by utilizing Python's inspect library to dynamically detect async functions and await them natively. Context Token Limits: Attempting to feed entire repo histories into the LLM caused token explosions. We resolved this by building a keyword router in code that filters search parameters before they are sent to the AI. Prompt Injection: Since incident reports are user-provided, we isolated raw text within explicit XML tags to ensure instructions were treated strictly as data. What We Learned: We learned the value of balancing deterministic code logic with generative AI. While LLMs excel at reasoning over raw data (such as parsing updates or finding similarities), critical operations like channel creation, permissions, and status updates are far safer and faster when written in strict, test-proven Python. What's Next: We plan to expand our integrations to include PagerDuty and Grafana MCP servers for on-call rotation management and live metrics monitoring, alongside integrating a vector store to allow the bot to recommend runbooks from similar past incidents.
Log in or sign up for Devpost to join the conversation.