Slack Compass

An AI agent that finds what's missing, forgotten, and contradictory in your Slack — before it breaks a launch, wastes a quarter, or reaches a customer.

Comment

Inspiration

Organizations don't fail because information is unavailable — they fail because nobody notices what's absent. The security reviewer who wasn't added to the launch thread. The decision made eight months ago that everyone quietly relitigates. The two teams telling customers opposite things about the same feature.

Summarizer bots and search tell you what was said. Nothing tells you what's missing. We kept watching good teams ship the wrong thing — not for lack of data, but for lack of a system that notices the gap. So we built an agent whose entire job is to detect absence, and to do it with cited evidence rather than vibes.

What it does

Slack Compass is an autonomous organizational-intelligence agent. It runs three detectors that answer questions no dashboard can:

  • Ghost Stakeholder (Missing) — Who should be involved but isn't? Finds the person who reviewed every prior payments launch but is absent from this one, with links to the messages that prove it.
  • Thread Resurrection (Memory) — Did we already decide this — and is that decision still valid? Resurfaces the prior decision, labels it CURRENT or STALE by age, tracks what it superseded, and returns a REUSE / REOPEN verdict when an assumption has changed.
  • Contradiction (Consistency) — Does the org contradict itself? Catches sales telling customers one thing while engineering says another, graded by severity, with both messages cited.

It ships as a native AI app on Slack's Agent messaging experience: DM it in natural language, get proactive warnings when launch messages or customer-facing claims appear, watch an entity continuously (it alerts only on new risk and tells you when the risk clears), and read live org health from the App Home dashboard. Every claim links to a real message permalink — no invented evidence.

How we built it

  • Slack Bolt + Socket Mode (TypeScript, ESM, Node 20+) as the agent runtime — no inbound webhooks, dials out from anywhere.
  • Agent messaging experience (agent_view) with suggested prompts, live "is searching…" status, and thread titles; DMs handled via message.im, channel questions via @-mention.
  • Real-time search API (assistant.search.context) collects live evidence from public channels on every analysis, with a search.messages fallback and an offline seed corpus so it's always demoable.
  • An evidence graph of people, claims, decisions, and domains, built from the retrieved messages.
  • A deterministic confidence scorer and rules-based labeler — the parts that assign risk are code, not a model, so they're reproducible.
  • A Claude reasoning layer (Anthropic API, claude-opus-4-8, structured JSON-schema output) grounds a plain-language insight and recommendation only in the retrieved evidence — and the whole layer degrades gracefully to deterministic output when no API key is present.
  • Block Kit "Compass Cards," a Watch Engine with an alert→resolve lifecycle, a health-check server, and graceful shutdown for hosting through the judging window.
  • 35 automated tests cover the detectors, scoring, evidence filtering, and card rendering.

The guiding principle: probabilistic where it helps (extraction, phrasing), deterministic where it matters (risk labels, citations).

Challenges we ran into

  • Slack's Agent messaging experience is brand new (June 2026). We migrated from assistant_view to agent_view, discovered that assistant_thread_started no longer fires and DMs arrive as plain message.im events, and rewired the whole Q&A surface accordingly.
  • A real-time-search feedback loop. With include_bots: true, the search API returned Compass's own posted cards as evidence — so a card saying "@alice is absent" was read back as "@alice was mentioned," inverting the finding on the next run. We had to teach the collector to recognize and exclude its own artifacts and non-seeded bot output.
  • Enterprise Grid sandbox quirks. auth.test returns an E… org id, not a T… workspace id, so channel APIs needed an explicit SLACK_TEAM_ID; listing private channels tripped a groups:read scope we didn't need; we scoped everything to public channels.
  • Detecting absence from hand-seeded data. When one account posts everything with today's timestamps, author/recency heuristics collapse — so involvement is derived from @-mentions and scoped by topic, not time.
  • Silent hangs. An un-timed-out live search call could stall a slash command forever; we added fail-fast timeouts that always fall back to seed so a command never goes dark.
  • Trustworthy confidence on tiny corpora without either bottoming out or overclaiming.

Accomplishments that we're proud of

  • It genuinely detects absence — the thing no summarizer does.
  • Every insight cites a real permalink. The model is never the source of truth for evidence.
  • Uses two of the challenge's qualifying technologies — Slack AI capabilities and the real-time search API — plus a grounded Claude reasoning layer.
  • Graceful degradation end to end: no API key → deterministic reasoning; live search stalls → seed fallback; runs fully offline for a reliable demo.
  • Decision lifecycle + supersession tracking — the memory detector reasons about whether a past decision is still valid, not just whether it exists.
  • A deterministic, tested core (35 tests) under an LLM-flavored product.

What we learned

  • Grounding every LLM statement in cited, retrieved evidence is the single biggest lever for trust in an agent.
  • Separating the probabilistic layer (extraction, narration) from the deterministic layer (labels, scoring, citations) is what makes an AI agent reliable enough to act on.
  • Building on a freshly released platform surface means the docs are the spec — and defensive engineering (timeouts, fallbacks, self-artifact filtering) matters more for a live judged demo than any extra feature.
  • "What's missing" is a fundamentally different and more valuable question than "what happened."

What's next for Slack Compass

  • MCP server integration to pull lifecycle records from external systems (Jira, GitHub, CRM) so validity checks reason over source-of-truth data, not just chat.
  • Learned stakeholder graphs per workspace, so expected-involvement isn't seeded but inferred.
  • Proactive digests — a weekly "here's what your org forgot / contradicted" summary.
  • Configurable domains and hints via a setup flow.
  • A Marketplace listing so any workspace can install it in one click.

Built With

  • agent-builder
  • agent-view
  • ai-agent
  • anthropic
  • app-home
  • assistant.search.context
  • block-kit
  • claude
  • claude-opus-4.8
  • esm
  • json-schema
  • knowledge-graph
  • llm
  • node.js
  • organizational-intelligence
  • real-time-search-api
  • slack
  • slack-ai-capabilities
  • slack-bolt
  • socket-mode
  • structured-outputs
  • typescript
Share this project:

Updates