CrossCheck

Your workspace already holds the answer — sometimes it holds two. Crosscheck finds where Slack disagrees with itself: contested decisions, both receipts, and books the meeting that settles it.

Comment

Inspiration

Every retrieval system in the knowledge-management space — Slack AI search included — is built to return one confident, cited answer. Coherence is the design goal. Which means when a workspace contains two conflicting decisions, the ranker silently picks one and presents it with citations. The team ships Friday and holds the release, depending on which channel you asked.

Crosscheck inverts the premise: conflicting retrieval results are the signal, not noise to be ranked away.

What it does

Mention @crosscheck with a topic. It searches the workspace through Slack's Real-Time Search and returns one of three verdicts:

  • ✅ ANSWERED — one decision, corroborated, current. Answer + receipts. No meeting needed.
  • ⚠️ CONTESTED — two opposing decisions, neither supersedes the other. Both sides, both owners, both permalinks — and a one-click action that books the reconciliation through MCP, with a pre-read canvas containing both threads.
  • ❓ UNANSWERED — nothing public covers it. It says so instead of hallucinating, and sets up a working session with a generated agenda.

The property everything hangs on: the LLM never emits the verdict. It emits structured claims (proposition, stance, subject — strict JSON). A deterministic function computes the verdict from those claims. An ANSWERED verdict without a permalink is structurally impossible: uncited claims are ineligible as evidence, and the boundary asserts it.

How we built it

Bolt for JavaScript (Socket Mode), one pipeline per inquiry:

  1. Budget guard — hard cap of 4 Slack retrieval calls per inquiry; it throws rather than risk rate-limiting the workspace.
  2. Query expansion (LLM) — 3 variants in workspace vocabulary. The mode is detected at boot via assistant.search.info: natural-language questions in semantic mode, synonym-heavy clauses in keyword mode.
  3. 3× assistant.search.context (RTS) with the event's short-lived action_token — used inside the handler, never persisted.
  4. Reciprocal Rank Fusion (k=60) across the three result lists, plus an author-identity filter (see "what we learned").
  5. Claim extraction (LLM, strict JSON) — decisions only; opinions, questions, and status updates are discarded.
  6. Deterministic verdict function — clustering by subject, stance opposition, same-thread supersession via permalink thread-roots, cross-channel divergence, staleness decay (confidence halves every 45 days).
  7. Block Kit verdict cards with receipt-tethered citations, and an MCP action layer: a genuine MCP client session to mcp.slack.com that creates the pre-read canvas (slack_create_canvas) and invites both owners (slack_send_message).

Both required technologies are load-bearing: strip RTS and there's no workspace context; strip MCP and it's a search box, not an agent.

Challenges and what we learned about the platform

  • RTS flags any app-posted message as is_author_bot: true — even chat.postMessage with a real user's xoxp token. Our bot filter silently deleted the entire seeded corpus on first contact. The fix ships in the product: filter by author identity (the agent's own user id + configurable CI-bot list), never the raw flag.
  • The agent's own replies echo back through context_messages of nearby results — quoted decisions were being re-attributed to humans. Evidence now excludes anything the agent said and anything said to the agent.
  • Indexing latency is real but short (~4 minutes measured): "real-time" search is minutes, not seconds — which shaped our seeding/testing workflow.
  • Retrieval is exact-ish even with AI search enabled, so the expansion layer matters in semantic mode too, not just keyword mode.
  • Supersession is the difference between a product and a toy. A thread that reversed itself is resolved. Detecting that deterministically (thread roots from permalinks + timestamp ordering) is what lets Crosscheck say "this was decided, then explicitly reversed — here's the current answer."

Accomplishments

  • The 10-second A/B a judge can reproduce unscripted: Slack search's one confident answer vs Crosscheck's CONTESTED card on the same workspace.
  • 37 tests including the two non-negotiables: SUPERSEDED → ANSWERED (not CONTESTED), and ANSWERED-without-citation is a thrown error, not a result.
  • A retrieval budget that respects the platform: ≤4 calls per inquiry, enforced structurally.

What's next

The Conflict Register: a dashboard of every unresolved contradiction currently sitting in the workspace — both permalinks, both owners, age. The verdict store already accumulates them; the surface is the next sprint. Beyond that: resolution tracking (Crosscheck noticing when a CONTESTED topic gets settled and closing the loop in both channels).

Built With

Share this project:

Updates