Devil's Advocate

The one bot in Slack that disagrees with you — it catches premature consensus and red-teams the decision with evidence before your team commits.

Comment

Inspiration

Every team has lived this: someone proposes vendor X, three people reply "sounds good," and thirty seconds later it's decided. Nobody wanted to be the person slowing things down. That's groupthink, and it's expensive - the worst decisions are often the ones that felt the most agreeable in the moment. We wanted the one voice that's structurally missing from most Slack channels: a fair, well-prepared dissenter that isn't afraid to be the "no" in a room full of "yes." Not a contrarian for its own sake - a red team on retainer.

What it does

Devil's Advocate quietly reads the flow of a channel. When it detects the team rapidly converging on one concrete decision - a run of agreement with no real pushback - it jumps into that decision's thread with a structured Block Kit message:

  • a one-line restatement of the decision it's about to challenge,
  • exactly three substantive objections/risks - not nitpicks, each one something that could actually change the call,
  • a closing note that makes its intent explicit: "Not trying to kill it - just want you deciding with your eyes open."

It's deliberately disciplined about when it speaks:

  • Confidence-gated: it only fires when its convergence analysis clears a high-confidence threshold, so casual chat, open debate, and Q&A are left alone.
  • Cooldown + anti-spam: a per-channel cooldown and an in-flight lock mean it posts at most once per decision and never dogpiles a thread.
  • Per-channel opt-in/out: /devil on, /devil off, /devil to check status - teams stay in control.

How we built it

  • Slack-native, no public server: built on Slack Bolt in Socket Mode, so it installs into a workspace with zero inbound webhooks or public URL - a big deal for security-conscious teams.
  • Clean separation of concerns: all decision logic lives in core.py with zero Slack SDK dependency - it's fully unit-tested (test_core.py) without tokens or network. app.py is a thin Bolt wiring layer.
  • Evidence-grounded via MCP: the objections aren't model recall - they're grounded in real external evidence. gather_evidence() acts as an MCP client: it launches a local web-search MCP server over stdio (the no-API-key DuckDuckGo MCP server, started with uvx), searches for the risks/downsides of the exact decision, then feeds those real results to the LLM so each of the three objections cites a source URL. If the MCP search ever fails, it degrades gracefully to LLM-only recall so the bot never goes silent.
  • Two-step LLM pipeline: detect_convergence() classifies whether the recent rolling window (last 20 messages/channel) shows rubber-stamping and returns a structured JSON verdict + confidence; gather_evidence() then runs the MCP-search-then-reason step to produce exactly three sourced counter-arguments. Reasoning runs on Gemini 2.5 Flash on GCP.
  • Concurrency-correct: Slack dispatches events on a thread pool, so a burst of "sounds good!" messages can enter the pipeline simultaneously. A single gate lock does an atomic enabled/cooldown/in-flight check-and-claim before the slow LLM calls, so exactly one dissent is posted per consensus burst (we actually hit and fixed a "3 dissents per decision" race).
  • Structured UX: the dissent is rendered in Block Kit - restatement, divider, three numbered risks, context footer - so it reads as a considered red-team memo, not a wall of text.

Challenges we ran into

  • Knowing when to speak - and when to shut up. Tuning convergence detection so it fires on real rubber-stamping but stays quiet during healthy debate was the core design problem; we solved it with a confidence threshold plus a "still-open debate = not converging" rule in the prompt.
  • The concurrency race. Multiple agreement messages arriving at once each passed the cooldown gate before the first dissent posted, producing triple replies. Fixed with an atomic gate lock + in-flight set claimed before the LLM calls.
  • Being disagreeable without being a jerk. The tone had to be firm but collegial - challenging the decision, never the people.

Accomplishments that we're proud of

  • A genuinely novel Slack agent - almost every bot exists to help you go faster; this is the only one whose job is to make you pause and think.
  • Testable, concurrency-safe architecture with logic decoupled from Slack.
  • A polished, restrained Block Kit UX that respects the channel.

What we learned

  • Convergence/groupthink is surprisingly detectable from message flow alone.
  • In a chat product, restraint is a feature: the value is as much in the 99% of the time it stays silent as in the 1% when it speaks.

What's next for Devil's Advocate

  • Add the Slack Real-Time Search API so objections also cite the team's own past decisions and threads ("your team already debated this in March"), alongside the live external web evidence MCP already provides.
  • Adjustable sensitivity (/devil sensitivity 1-3), per-topic muting, and a weekly "decisions we flagged" digest.

Built with

Slack Bolt, Socket Mode, Block Kit, Python, MCP (Model Context Protocol - DuckDuckGo web-search server over stdio), Google Gemini 2.5 Flash, GCP.

Built With

Share this project:

Updates