FaultMaven for Slack

Run the whole troubleshooting investigation in the Slack thread where the problem already lives - a copilot that works a problem like a seasoned engineer and turns every fix into a runbook.

Comment

Inspiration

When production breaks, the war room is a Slack thread. But the actual troubleshooting isn't — it's scattered across a dashboard in one tab, a runbook wiki in another, and the one engineer who remembers the last time this happened. So the thread fills with "can someone check the logs?" while the real work happens somewhere else. And the moment the incident closes, everything the team just learned evaporates — the next outage starts from zero.

We wanted the investigation to live where the problem already does. Not a bot that pastes a canned answer, but something that actually works the problem in the thread, with the people already there.

What it does

FaultMaven for Slack runs a real troubleshooting investigation inside your thread. You @mention it on a problem — a broken deploy, a failing service, a config that isn't behaving — or run Ask FaultMaven from the ⋮ menu on any alert, error, or stack trace. It triages the symptom, forms a hypothesis, and asks for the specific logs or configs that would confirm or rule it out. You reply with the evidence; it reads it, grounds its reasoning in your runbooks and past fixes, and keeps going until the root cause is found and the fix is verified. When the case resolves, the fix is captured as a runbook — so the next time it happens, the investigation starts from what the last one learned.

Two decisions make it safe to drop into a shared channel: it acts only on threads you summon it into (no ambient listening, no reading channel chatter), and it won't fabricate a root cause — while a case is open it names the evidence it still needs instead of posting a confident guess. It suggests; you approve and execute.

How we built it

The Slack agent is a thin, careful bridge over an existing investigation engine. Every turn maps to the FaultMaven REST API: a Slack thread becomes a case (POST /cases), and each message becomes a turn (POST /cases/{id}/turns, multipart — so attached logs ride along as evidence). The backend never sees a Slack token or payload; the agent is the only thing that touches both sides.

  • Bolt for Python, built around Slack's Assistant / AI-App container so the side panel gets native suggested prompts, live status, and streamed replies — plus app_mention + message shortcuts for war-room use in a channel.
  • Multi-workspace OAuth over HTTP — a FastAPI host serving /slack/events, /slack/install, and /slack/oauth_redirect, with per-workspace bot tokens in Postgres and a dependency-free /health for the platform.
  • One thread = one case, summon-to-create then auto-continue on reply, with a one-turn-per-thread gate so a busy investigation never double-runs.
  • Deployed on Kubernetes behind slack.faultmaven.ai, installed across multiple live workspaces.

Challenges we faced

  • The engine is honest, which makes the UX harder — in a good way. FaultMaven is built on a no-incorrect-conclusion guarantee: it will not declare a resolution it can't support. That meant the agent had to render "here's the evidence I still need" as a first-class state, never paper over an open case with a plausible-sounding answer.
  • Transport was a real rebuild. The first cut ran on Socket Mode — fine on a laptop, but the App Directory and multi-workspace installs require public HTTP events and OAuth distribution. We rebuilt the transport layer end to end without touching the turn pipeline.
  • Trusting a non-deterministic backend safely. Gateway timeouts and dropped connections can mean a turn maybe committed. Telling a user "try again" when the backend already ran the turn would double-run the investigation, so we classified those cases as indeterminate and surfaced them honestly instead of guessing.
  • Making it production-safe under load — scheme-gating rendered links so LLM output couldn't inject Slack broadcasts, moving backend startup off the liveness path so a slow backend couldn't crash-loop the pod, and locking the shared auth token so concurrent turns couldn't blank each other out.

What we learned

The hard part of an AI agent isn't the happy path — it's the failure modes the user sees. A dropped turn, a stuck thread, a wall of text, a confident wrong answer: those are what break trust. Designing for "what does this look like when it goes wrong, in front of the whole channel?" shaped more of this project than the feature list did. The discipline of never rendering an unsupported conclusion turned out to be a product strength, not a constraint.

What's next for FaultMaven for Slack

Per-user identity linking so a case knows who asked, and team- and org-scoped visibility that mirrors how Slack workspaces and channels already map to teams and organizations. Beyond that: pulling resolved cases into the shared knowledge base automatically — so the flywheel spins without anyone thinking about it — and tightening the war-room surface (threaded digests, handoffs between responders) for larger incidents.

Built With

Share this project:

Updates