Inspiration
We've all watched it happen. The one engineer who really understands the deploy pipeline goes on vacation — and suddenly nobody can ship. The teammate who set up billing leaves, and a month later an invoice bug takes a week to fix because the knowledge walked out with them.
That's bus factor: the number of people who'd have to disappear before a project stalls. For the things that matter most, that number is often 1 — and nobody notices until it's too late.
We wanted a Slack agent that could see this risk before it bites. Not another search bot or Q&A bot, but something that actually reasons about where knowledge is concentrated and where it lives only in chat, undocumented. And because Slack's terms forbid storing message content, we set ourselves a hard constraint: do all of this without ever keeping a single message.
What it does
SignalKeeper detects two kinds of invisible risk in a Slack workspace:
- Bus-factor risk — a topic whose expertise is concentrated in one person (a single point of failure).
- Knowledge islands — topics discussed constantly in chat but never documented.
It exposes three surfaces, all in Slack:
/busfactor [topic | #channel]→ a ranked risk report (SPOFs, islands, and a recommended fix per risk).@SignalKeeper who knows about <topic>?→ the expertise ranking for a topic plus its bus factor, in-thread, with a live "thinking…" status.- App Home dashboard → a workspace risk summary with a one-click "Run a scan" button.
When you approve a recommendation, it acts: posts the report, opens a thread tagging the knowledge holder, or auto-creates a "knowledge-capture" canvas to get the expertise written down.
How we built it
We built SignalKeeper as a clean, hexagonal (ports & adapters) architecture, in seven test-first phases, ending at 100% test coverage with strict typing and CI.
The heart of it is a deterministic reasoning core — pure functions with zero IO. For each topic, we compute a recency-weighted expertise score per person from substantive contributions, resolved answers, and inbound @mentions, where an event $\Delta t$ days old is weighted by an exponential half-life $H$:
$$w(\Delta t) = 0.5^{\,\Delta t / H}$$
We then measure how concentrated the knowledge is with two indices — the Herfindahl-Hirschman Index and the Gini coefficient — over each contributor's share $s_i$ of the total expertise:
$$\mathrm{HHI} = \sum_{i} s_i^{2}, \qquad G = \frac{\sum_i \sum_j |x_i - x_j|}{2\,n \sum_i x_i}$$
Finally, the bus factor is the smallest number of top contributors whose removal drops the remaining coverage below a threshold $\tau$ (default $50\%$):
$$\text{bus factor} = \min\Big{\,m : 1 - \tfrac{\sum_{i\le m} x_{(i)}}{\sum_i x_i} < \tau \,\Big}$$
A bus factor of $1$ is a single point of failure — the headline risk.
Around that core:
- Slack Real-Time Search API retrieves conversations on demand.
- An LLM behind a port classifies messages into topics/kinds (we made it provider-agnostic — OpenAI or Anthropic via one config switch).
- NetworkX builds an ephemeral expertise graph, rebuilt per analysis and never saved.
- A privacy boundary (
build_domain_messages) copies only metadata inward; raw text is dropped on the floor and never persisted or logged — enforced by an automated privacy-guard test. - A fixture-replay mode records Real-Time Search payloads so the whole agent runs reproducibly, even without a paid Slack search tier.
Challenges we ran into
This project was a steady stream of real-world platform reality checks:
- The RTS API is tier-gated. Slack's semantic search needs a Business+/Enterprise+ workspace. We turned that constraint into a feature: a fixture-replay search adapter that serves recorded payloads, so the demo is reproducible anywhere.
- An async event-loop bug at launch. The Socket Mode handler builds an
aiohttpsession in its constructor, which callsasyncio.get_running_loop(). Constructing it beforeasyncio.run()started the loop threwRuntimeError: no running event loop. The fix: construct it inside the coroutine — and we added a regression test so it can't come back. - Anthropic ran out of credits mid-demo. Because the LLM sat behind a port, swapping to OpenAI was a new ~40-line adapter plus a config switch — nothing else changed. That was the architecture earning its keep.
- The Slack MCP server rejected our token.
mcp.slack.comrequires its own OAuth flow and returned HTTP 400 to our user token at theinitializehandshake. Rather than block the demo, we built a Slack Web API action provider (the default) that posts threads and creates canvases with the bot token, and kept MCP as an opt-in. - A dozen small Slack-platform gotchas: the app manifest rejecting a custom
_metadatakey, slash commands not registering until the manifest is pushed and the app reinstalled, RTS not exposing thread-resolution data, and more.
Through all of it, we held the line on the non-negotiable: no raw message content is ever persisted or logged.
Accomplishments that we're proud of
- A genuine reasoning core, not a chatbot — real bus-factor math, HHI + Gini, ranked risks.
- 100% test coverage, mypy strict on the core, green CI — production-quality, not a hack.
- Privacy-by-design that's actually tested: a guard test runs a full analysis and asserts a sentinel message string never reaches persistence, logs, the report, or any action.
- Provider-agnostic by design: we swapped the entire LLM provider under deadline pressure in minutes, and the action layer the same way — proof the ports & adapters weren't just for show.
- All three Slack technologies integrated (Agent runtime, Real-Time Search API, MCP), plus a reproducible demo that doesn't depend on a paid tier.
What we learned
- Clean architecture pays off when it's tested under fire. The LLM and action swaps weren't theoretical — we did them live, and the seams held.
- Designing around a hard constraint makes the design better. "Never store message content" forced the metadata-only domain model that became the cleanest part of the codebase.
- The Slack platform has real edges: Socket Mode vs. OAuth, RTS tiers, MCP's OAuth, manifest validation, slash-command registration. We learned them the hard way so the next build won't.
- Turning fuzzy risk into math is powerful. Expressing "knowledge concentration" as HHI, Gini, and a coverage-threshold bus factor made the agent's judgments explainable and deterministic.
What's next for SignalKeeper
- Thread-resolution signals — infer "answered & resolved" from reactions and
conversations.replies, since RTS doesn't expose it. - Proactive alerts — scheduled scans that DM an owner the moment a new single point of failure appears.
- Trend tracking — persist derived metrics over time to show whether bus factors are improving (the storage port already keeps content-free summaries).
- Graph-based bus factor — a min-cut over the expertise graph to complement the coverage-threshold method.
- Full Marketplace distribution — wire the OAuth install flow so any organization can add SignalKeeper in one click.
Log in or sign up for Devpost to join the conversation.