Slack Health Agent

No human needed to notice the problem first.

How I built it

The architecture has three layers:

1. Slack Bot — FastAPI + Slack Bolt (async). Handles slash commands and receives Alertmanager webhooks on /alert.

2. MCP Server — A FastMCP server exposing three Prometheus tools (get_service_health, get_top_errors, get_latency). Claude connects to this over SSE and decides which tools to call based on the prompt.

3. Observability stack — HotROD (a real Go microservice) running on GKE Autopilot, scraped by Prometheus via a ServiceMonitor CRD. Alert rules defined as PrometheusRule resources. Alertmanager configured to webhook the bot.

The key insight: Claude sits between Slack and Prometheus. It doesn't just format data — it reasons about it. The same MCP architecture works for any service by changing one env var (SERVICE_NAME).

Challenges

GKE Autopilot restrictions — Autopilot blocks node-level access, which broke kube-prometheus-stack's node exporter and kube-system scrapers. Had to disable five Helm components to get Prometheus running.

Slack's 3-second timeout — Claude + MCP + Prometheus takes 5-10 seconds. Solved with await ack() immediately followed by await respond() — Slack gets its 200 response in time, the answer comes as a follow-up.

Alertmanager config with Prometheus Operator — The Operator merges your config with its defaults, which reference a null receiver. Took significant debugging to understand that your secret must define the null receiver too or the Operator rejects the entire config.

MCP SSE host binding — FastMCP defaults to 127.0.0.1, which refuses connections from other pods. The fix was host="0.0.0.0" in the FastMCP() constructor — not in mcp.run().

What I learned

• How MCP enables LLMs to use tools over a network — and why SSE transport is necessary in a distributed system
• How the Prometheus Operator works — ServiceMonitor, PrometheusRule, and Alertmanager CRDs as Kubernetes-native config
• The difference between pull-based and push-based observability, and why both matter for production support
• How to build async Python services that handle concurrent Slack requests without blocking

What's next

• Add Cloud Logging as a second MCP tool — Claude would correlate metrics and logs in one response
• Add distributed tracing (Cloud Trace / Jaeger) as a third tool — full observability stack in one agent
• Persistent incident context — Claude remembers what was discussed in the last 30 minutes so engineers don't repeat themselves

Built With

• alertmanager
• artifact
• autopilot
• bolt
• claude
• docker
• fastapi
• gke
• google
• hotrod
• httpx
• kube-prometheus-stack
• kubernetes
• mcp
• prometheus
• python
• registry
• slack