Shadow Incident Engineer

💡 Inspiration

During a high-severity production outage, every second counts. Traditional incident response is plagued by chaotic context-switching: engineers must scramble between log aggregators, terminal windows, version control history, and team chats just to figure out what broke.

Instead of building another fragmented third-party web dashboard that forces engineers to leave their workspace and log into a separate portal, we asked a simple question: What if the dashboard came directly to the communication layer?

We built Shadow Incident Engineer (SIE)—a 100% Slack-native incident operations agent designed to consolidate the entire diagnostic lifecycle into a single interactive Slack thread where your team is already talking.

🛠️ What it does

SIE acts as an autonomous, real-time diagnostic partner when critical failures strike:

1. Intercepts Outage Alerts: Seamlessly listens to raw application monitoring alerts over secure WebSockets using Slack Socket Mode.
2. Generates Interactive Block Kit Diagnostics: Instantly constructs a data-dense diagnostic engine card directly in your Slack channel.
3. Pulls Live System Logs: Communicates with a custom Model Context Protocol (MCP) Server to securely extract raw system trace logs.
4. Traces Version Control History: Automatically maps the failure signature to your Git repository, pulling the last three commits to pinpoint exactly who deployed what.
5. Surfaces Tribal Knowledge: Uses the Slack Real-Time Search API to programmatically crawl workspace archives and surface deep-links to past identical incidents, eliminating redundant troubleshooting.
6. Enables Instant Action: Supports inline interactive state changes (like "Assign to Me") natively inside the conversation stream.

🏗️ How we built it

We engineered SIE using a modern, event-driven TypeScript backend.

• Slack SDK & Socket Mode: Leveraged WebSockets to ensure highly responsive, bi-directional communication between our agent and Slack, entirely bypassing the need for public-facing HTTP request endpoints.
• Model Context Protocol (MCP): Implemented an MCP server configuration to securely expose specific contextual data layers (server log outputs and system environments) directly to our LLM runtime.
• Block Kit Builder Engine: Custom-designed a dynamic, compact layout engine to surface nested technical telemetry inside Slack's UI limits.
• Slack Search API integration: Programmed custom query-parsing wrappers to instantly crawl historical channel data.

🛑 Challenges we ran into

• Displaying Dense Data in Slack UI: Fitting traceback errors, Git commit hashes, and log files into a standard chat stream without causing "wall-of-text" fatigue was a major design challenge. We solved this by strictly grouping metadata into collapsible/structured Slack Block Kit elements.
• Securing MCP Boundaries: Safely executing system tasks (like reading logs) via our MCP server required setting strict run-time boundaries so the LLM could query context without risking arbitrary command execution.

🎉 Accomplishments that we're proud of

• Zero Dashboard Overhead: We are proud of our deliberate architectural choice to skip a traditional web interface. Keeping engineers focused entirely inside Slack preserves absolute focus during high-stress outages.
• Frictionless Real-Time Sync: Connecting live error monitoring to Slack’s search index seamlessly bridges machine-level system bugs with human-level conversation history.

🧠 What we learned

Building SIE taught us how incredibly powerful Slack's application layer is when used as an operating system rather than just a messaging app. We learned how to write clean MCP-to-LLM bridge layers, optimize search API call limits, and leverage interactive payloads to run instant state updates without external web portals.

Built With

• ai
• git
• model-context-protocol-(mcp)
• node.js
• slack-api
• slack-block-kit
• typescript
• websockets