Tribal Knowledge Agent for Slack

• 

  Architecture

• 

  Tribal Agent Response

• 

  No Decision Found Response

• 

  Bot Token Scopes

Inspiration

Every team has an answer buried in some thread from months ago that never made it into docs. I've spent 3.5 years building RAG systems for enterprise clients, and the pattern is always the same: institutional knowledge lives in chat, not in a wiki, and by the time someone needs it, they either have to ping a person directly or dig through a wall of search results themselves. I wanted to see if I could get an agent to actually read that history and answer the question directly, instead of just pointing at threads.

What it does

I mention the bot in Slack with a question like "how do we handle rate limiting on the payments API?" and it searches the workspace's own message history, synthesizes a direct answer, and cites exactly which messages it pulled from. If the answer changed over time, it says so explicitly instead of picking one version silently. If two people disagreed and one approach won out, it surfaces that. And if there's genuinely no answer in the workspace, it says that too, instead of making something up.

How I built it

The core is Slack's Real-Time Search API (assistant.search.context), wired into a Bolt app over Socket Mode. When the bot gets an app_mention event, it grabs the action_token from the event payload and uses it to search live, no pre-indexing, no stored vector DB.

I hit a wall early: my sandbox workspace has is_ai_search_enabled: false, meaning RTS only does keyword search, not semantic. A natural question like "how do we handle rate limiting" shares zero literal words with a message that says "seeing 429s, going with exponential backoff." So I added a query expansion step: Claude turns the user's question into 2-3 keyword-style variants before hitting RTS, then I merge and dedupe the results.

Retrieved messages go to a second Claude call that synthesizes an answer with inline citations, explicit conflict-flagging when sources disagree, and a staleness check if the most relevant info is over 60 days old. The reply goes back via Block Kit with a status indicator ("searching...", "synthesizing...") while it works, and auto-added 👍/👎 reactions for feedback.

Challenges I ran into

The big one was discovering RTS defaults to keyword-only search on my sandbox tier. I initially assumed semantic matching would just work since the docs describe it as a feature of the API, and my first tests came back empty for perfectly reasonable questions. Checking assistant.search.info confirmed the workspace didn't have semantic search enabled, which meant building the query expansion layer wasn't optional polish, it was the fix for the actual retrieval gap.

I also had a subtler bug where results were technically correct but noisy: the bot's own test questions (with @mentions in them) were getting pulled back into search results and cited as sources, since they contained literal overlapping keywords with later real questions. Had to filter those out explicitly before merging.

Accomplishments that I'm proud of

The scenario that mattered most to me is the one where the agent correctly says "no decision has been made yet" instead of inventing an answer. Most RAG demos only show the happy path where the system confidently retrieves something. Getting it to be honest about the absence of an answer, and to flag when sources conflict rather than silently picking one, felt like the actual hard part of the project, not the retrieval plumbing.

What I learned

Real-time, non-indexed retrieval changes the tradeoffs I'm used to from building RAG pipelines with vector stores. There's no embedding step to fall back on for semantic matching, so you're at the mercy of whatever retrieval mode the platform actually gives you, and you have to design around that constraint rather than assume it away. I also got a much better feel for Slack's newer Real-Time Search API versus the legacy Data Access API, including the action_token lifecycle and the granular search:read.* scope model.

What's next for Tribal Knowledge Agent for Slack

I'd want to add persistent feedback storage (right now 👍/👎 reactions just log to console), wire in the native Slack Assistant panel UI for a more integrated experience, and test how the query expansion holds up across a much larger and messier real workspace instead of my seeded scenarios.

Built With

• anthropic-claude-sonnet
• claude-api
• javascript
• node.js
• slack-block-kit
• slack-bolt-sdk
• slack-real-time-search-api
• socket-mode