Inspiration
When a critical system goes down, the clock starts ticking. As the engineers frantically jump between PagerDuty to read the alert, Jira to check for active deployments, and GitHub to hunt down recent commits. We realised that if we could reduce analysis time to near zero, we could save companies massive amounts of downtime. We wanted to build an agent that lives where the developers already are—Slack—and handles context switching for them in seconds.
What it does
This project is a fully autonomous, cross-stack investigator. When an engineer is paged in Slack with a critical alert, they simply tag @incident_responder and provide the PagerDuty ID and Jira ID. Instead of just answering questions, the agent: 1. Instantly queries PagerDuty to understand the exact error payload. 2. Checks Jira for active deployment tickets related to the affected service. 3. Scans GitHub for recent code changes. 4. Synthesises all this context into a beautifully formatted, actionable root-cause Markdown report directly in Slack. It features a live, async Slack progress bar (… ⏳) so the on-call engineer knows exactly what tools the agent is executing in real-time.
How we built it
We built a highly optimised, single-container architecture deployed on Google Cloud Run. • Frontend/Routing: We used Slack Bolt (FastAPI/Uvicorn) to handle asynchronous webhooks. We implemented interceptors to gracefully handle Slack's aggressive 3-second retry loop, preventing message spam while the LLM thinks. • The Brain: We used the Google GenAI SDK (Gemini 3.5 Flash). We built a custom Agentic Loop that supports Parallel Function Calling, allowing the AI to hit multiple infrastructure APIs simultaneously. We routed the agent's behaviour using a strict SKILL.md playbook to enforce plain-text Markdown outputs over raw JSON. • The Connectors (MCP): This is where the magic happens. We utilised the Model Context Protocol (MCP) to bridge the LLM with our external tools. We ran the official Node.js GitHub MCP server alongside a custom Python FastMCP server (handling PagerDuty and Jira). • The Architecture: To maximise speed and security, we ran the MCP servers in the same container as the FastAPI bot. They communicate via stdio (Standard Input/Output) rather than HTTP, resulting in zero network latency and a fully atomic deployment lifecycle.
Challenges we ran into
Building a seamless bridge between open-source protocols and strict enterprise AI SDKs was incredibly difficult. We hit several major roadblocks: 1. The MCP Subprocess Crash: Running background terminal processes (stdio) inside a Docker container wiped out the $PATH and $PYTHONPATH environments. We had to write custom logic in our orchestrator to safely merge the container's system environment with our API keys to keep the background tools alive. 2. Strict OpenAPI Validation Walls: The open-source GitHub MCP server uses standard JSON Schema (including $schema tags), but the Gemini SDK uses Pydantic to strictly enforce OpenAPI 3.0 standards. We had to write a surgical, recursive schema sanitiser in our mcp_client.py to scrub incompatible metadata out of the payloads before the LLM would accept them. 3. Vertex AI Prefixes: We discovered an undocumented quirk where the Vertex AI backend invisibly wraps tools in a default_api: prefix, causing our routing logic to crash. We had to build a custom string-stripping layer to clean the namespace before executing the tools.
Accomplishments that we're proud of
1. True Agentic Reasoning: The agent doesn't blindly run a script. If a user asks to check Jira but forgets the ticket number, the agent realises it lacks parameters and gracefully asks the user for the ID instead of failing.
2. Parallel Execution: By handling multiple tool requests in a single execution turn, we brought the total time to synthesise a multi-platform incident report down from 5 minutes to ~2 minutes.
3. The Async UX: Using asyncio background tasks to edit the Slack message in real-time with an updating progress bar makes the bot feel incredibly polished and enterprise-ready.
What we learned
We learned the immense power of Prompt-Driven Routing. By injecting specific repository targets and formatting rules into a SKILL.md system prompt, we drastically reduced the agent's "hallucination rate" and prevented it from wasting execution turns randomly searching GitHub. We also learned how to deeply manipulate the Model Context Protocol to seamlessly blend Python and Node.js toolsets under a single LLM brain.
What's next for Cross-Stack Incident Investigator
Currently, the agent is a read-only diagnostic tool. The next step is Auto-Remediation. We want to expand our MCP servers to allow the agent to confidently propose a fix, ask the on-call engineer for approval via interactive Slack buttons, and automatically hit the GitHub API to revert the offending Pull Request.
Log in or sign up for Devpost to join the conversation.