Verity — Fact-Checking Agent for Slack

Verity watches Slack channels for misinformation and silently warns the poster before it spreads — or fact-checks anything you paste, with sourced, authority-ranked verdicts in seconds.

Comment

Inspiration

Every day, as I scroll through social media like Instagram, I see questionable claims and misinformation spreading rapidly. But there is no fast way to verify them in the moment. I tried sharing these reels and videos with ChatGPT and other LLMs, but they all said the same thing: they couldn't extract the exact captions or spoken details of the video.

I thought: there should be an app or agent where I can just share a link, and it automatically goes out, searches through credible sources, and returns a detailed summary of the claim. When I saw the Slack Agent Builder Challenge on Devpost, it clicked — why not build this exact solution inside Slack? This way, I could implement the idea I had and bring instant verification directly into the workspace where teams share and discuss content.

Verity is the fact-check button the internet doesn't have — one paste, one sourced answer, right inside Slack where your team already shares and discusses content.

What It Does

Verity works two ways. Proactively, it watches for links posted in channels it's in, quietly runs them through the full verification pipeline in the background, and — only if the claim comes back False or Misleading — sends a private, ephemeral warning to the person who posted it. Nobody has to ask. Nobody gets publicly called out. On demand, paste any link or claim directly and Verity replies in-thread with a sourced, source-quality-weighted verdict — True / False / Misleading / Unverifiable — in under 30 seconds.

Three input types supported for the on-demand flow:

  • Plain text claim — paste any factual statement directly
  • YouTube video link — Verity fetches the video transcript and checks the spoken claim
  • Article / news link — Verity extracts the article body and identifies the checkable claim

What you get back:

  • Verdict with confidence score (0–100%)
  • Summary of the key evidence
  • Sourced citations with authority tier badges (Tier 1 / Tier 2 / Tier 3)
  • A Slack Canvas full report (persists indefinitely)
  • An entry in the workspace's Slack Lists claim directory
  • Workspace memory — if this claim was previously discussed in Slack, those threads surface automatically

How We Built It

Verity is a 5-stage agentic pipeline built in Python with the Slack Bolt SDK, plus a standalone Node/Express MCP server for search.

Stage 1 — Ingestion

Detects input type (YouTube URL → youtube-transcript-api; article URL → trafilatura; plain text → passthrough). All failure paths return a safe error dict — no crashes, no ingestion fragility on the demo path.

Stage 2 — Claim Extraction

Gemini 3.1 Flash Lite identifies the specific checkable claim and classifies its type: single_fact, comparative (e.g., "X has more protein than Y and Z"), or causal. Comparative claims are classified distinctly so they aren't collapsed into a single-fact check, and the agent is prompted to investigate each compared entity separately.

Stage 3 — Agentic Verification Loop (MCP Integration)

Gemini autonomously formulates search queries and calls a self-hosted Brave Search MCP server (Node/Express, SSE transport, vendored in this repo at brave-search-mcp-sse/). Key safety features:

  • Hard 4-turn cap — prevents runaway agentic loops
  • Deduplication guard — identical queries are short-circuited with a synthesis nudge
  • No Automatic Function Calling — we use a manual tool loop to maintain full observability

Stage 4 — Verdict Synthesis

A structured synthesis turn (tools disabled, response_schema enforced) produces a typed JSON verdict: { verdict, confidence, summary, sources[] }.

Citation hallucination guard: A whitelist of every URL returned by Brave Search is passed to the synthesis prompt. A post-processing filter then structurally removes any citation not in the whitelist. Verity cannot fabricate sourced evidence — if search fails, verdict is forced to Unverifiable with confidence ≤ 0.30.

Stage 5 — Slack Delivery

Block Kit threaded reply (on-demand flow) or ephemeral Block Kit warning (proactive flow), color-coded verdict, tiered source badges, workspace memory section, and a Slack Canvas report link. Slack Lists logs every verdict for team-level moderation.

Authority Tier System

A named, deliberate design decision — not a default pass-through:

  • Tier 1 (0.75–1.00): .gov, .edu, .mil, PubMed, WHO, USDA, Nature, The Lancet, NEJM
  • Tier 2 (0.45–0.74): Reuters, AP, BBC, The Guardian, Snopes, PolitiFact, FactCheck.org
  • Tier 3 (0.10–0.44): General web, blogs, aggregators — included but weighted down

Why This Is an Agent, Not a Bot

Most "AI Slack tools" are request/response: you ask, it answers. Verity's core loop is different — it runs unprompted. The proactive scanner background-scans every channel message it can see, and if a claim resolves to False or Misleading, it makes an autonomous decision to intervene: not a public callout, which would embarrass the poster and likely get the feature muted by admins, but a quiet, ephemeral, single-recipient warning. That's a judgment call about how to act, not just whether to act.

Two more deliberate agent-design choices worth naming:

  • Manual tool loop, not automatic function calling — every search call and result is inspectable, bounded by a hard 4-turn cap and a deduplication guard against runaway agentic loops.
  • Structural, not prompt-only, safety — the citation whitelist and the forced-Unverifiable-on-search-failure path are enforced in code after the model responds, not just requested of the model.

Challenges We Ran Into

Instagram/TikTok ingestion — We tested it on Day 1. Video download works, but Instagram exposes no transcript or caption API of any kind — confirmed against yt-dlp's own issue tracker. Getting spoken-word transcripts would require Whisper STT: ~10–30s latency, ~2GB model dependency, and a live-demo risk we weren't willing to take. Deliberately scoped out.

Agentic loop event loop conflicts — The Slack Bolt app uses threads; the MCP client is async. We built a persistent background event loop (mcp_client.py) with a dedicated thread and run_coroutine_threadsafe to bridge sync and async cleanly — a genuinely non-trivial concurrency problem, now fully covered by its own test suite (success, reconnect-after-failure, timeout, and idempotent shutdown).

Hallucinated citations — Early versions of the synthesis prompt let Gemini invent plausible-looking URLs. We fixed this structurally: every URL retrieved by Brave Search goes into a numbered whitelist in the synthesis prompt, and a post-processing filter enforces it structurally.

Web-evidence dependency — Verity gathers all web evidence via a Brave Search MCP server over SSE. If that server is unreachable, it returns an honest "Unverifiable" verdict (confidence capped at 0.30) with no fabricated sources rather than guessing from parametric knowledge.

Accomplishments We're Proud Of

  • Anti-hallucination architecture — Two-layer protection (whitelist in prompt + structural post-processing filter) means Verity's citations are provably grounded in real search results
  • Source authority as a named design decision — The score_authority() function is explicit, inspectable, and testable — not buried middleware
  • Comparative claim handling — Comparative claims are recognized and handled as multi-entity comparisons rather than single facts
  • Proactive, socially-aware misinformation catching — Verity acts on its own initiative and chooses non-disruptive, private delivery
  • Full test suite — All 6 pipeline modules/components have pytest tests with fixed inputs/outputs (66 total tests passing). Line coverage on the fast/mocked suite is 67% (pytest -m "not slow"); network-dependent live-API paths (YouTube/article fetching, live search) are covered separately under tests marked slow rather than skipped silently.
  • Graceful degradation — Every failure path produces a clean, honest UX — never a confident verdict from ungrounded knowledge

What's Next for Verity

  • Twitter/X post ingestion
  • Multi-claim documents — long articles with many claims
  • Team notifications — alert teammates when a claim has already been debunked
  • Workspace-level claim history dashboard
  • Fine-tuned claim extraction for edge cases

Track

Slack Agent for Good — addressing misinformation as a public health problem. Focus areas: public health, education, media literacy.

Built With

  • asyncio
  • brave-search-api
  • express.js
  • github
  • google-gemini
  • mcp-(model-context-protocol)
  • node.js
  • pydantic
  • pytest
  • pytest-cov
  • python
  • server-sent-events-(sse)
  • slack-assistant-sdk
  • slack-block-kit
  • slack-bolt-sdk
  • slack-canvas-api
  • slack-lists-api
  • slack-real-time-search-api
  • slack-workflow-builder
  • trafilatura
  • typescript
  • youtube-transcript-api
Share this project:

Updates