TruthExpiry

TruthExpiry prevents stale Slack answers by using real-time search and MCP evidence to find the current valid answer, explain what changed, and ask the right owner to confirm it.

Comment

Inspiration

Slack agents often sound confident because search returns messages that look relevant. Semantic similarity is not proof a claim is still true — a thread from last quarter can read like an answer today.

TruthExpiry started from one question: how do you stop an agent from repeating stale information without asking the LLM to be the judge?

The insight: separate what someone said in Slack from what the organization's lifecycle records say now. Search finds candidates; authoritative records decide validity.

Similarity is not validity.

What it does

TruthExpiry is a Slack agent that verifies concrete claims against lifecycle evidence and returns a deterministic verdict:

  • CURRENT — matches authoritative records
  • SUPERSEDED — a newer record replaced the value
  • CONFLICTING — multiple authoritative sources disagree
  • UNVERIFIED — no matching authority (fail-closed, not guessed)

Flow: User @mentions the app or sends a DM → RTS (assistant.search.context) retrieves public-channel evidence → OpenAI extracts zero or one structured claim → deterministic code fetches lifecycle evidence and assigns the label → Slack shows the verdict with proof links (e.g. PROD-482, PROD-511).

Examples from the demo:

  • "Is report export available on Starter?" → SUPERSEDED (was enabled, now disabled)
  • "Is the API rate limit 100 for Starter?" → SUPERSEDED (now 50)
  • "Tell me about report export" → guidance only — no invented validity label

Live authority: Lifecycle evidence comes from a Streamable HTTP MCP server or GitHub Issues with te:* labels — same pipeline, swappable adapter. Edit a GitHub label and the verdict flips without redeploying.

How I built it

Stack: Python 3.10+, Slack Bolt (Socket Mode), assistant.search.context RTS, OpenAI (gpt-4.1-mini) for extraction only, MCP Streamable HTTP for lifecycle evidence, Docker, GitHub Actions CI.

Architecture: Ports-and-adapters design with a strict boundary:

  • Probabilistic: LLM extracts structure (entity, attribute, scope, stated value) from bounded query text and opaque evidence IDs
  • Deterministic: catalog grounding, lifecycle fetch, and label assignment — the model never decides CURRENT/SUPERSEDED/CONFLICTING/UNVERIFIED

Key engineering choices:

  • One RTS call per request; public channels only (search:read.public)
  • No persistence of Slack message bodies across requests
  • Bearer-authenticated MCP; health/readiness probes (/healthz, /readyz)
  • Worker stays alive during temporary MCP outage; readiness recovers automatically
  • 540+ automated tests — unit, integration, container smoke, dependency audit

Deployment: Worker on Render (GitHub Issues mode for judge uptime); optional lifecycle MCP for full two-service topology.

Challenges I ran into

Drawing the LLM boundary. It's tempting to let the model "decide if something is still true." That's fragile and unauditable. The hard part was keeping extraction useful while making validity 100% deterministic and testable.

RTS + lifecycle timing. Health /readyz can return 200 before the MCP HTTP port is listening — caught a flaky CI race and fixed it with proper port readiness polling.

Fail-closed vs helpful. Vague questions ("what is the rate limit?") must not get a validity label invented. Building query grounding and a claim-schema catalog so the agent guides users instead of guessing was a real UX + safety tradeoff.

Socket Mode ops. Only one active connection per app token — local dev and Render can't run simultaneously. Documented clearly for judges and demo stability.

Scope discipline. Public-channel evidence only, no cross-turn memory, no Slack content retention — constraints that simplify privacy but required careful pipeline design.

Accomplishments that I'm proud of

  • A clear, memorable principle: similarity is not validity — implemented, not just marketed
  • End-to-end working demo: SUPERSEDED, CURRENT, safe refusal, numeric validation, and live GitHub authority flip
  • Mature hackathon repo: ports/adapters, 540+ tests, CI green, Docker, runbooks, Render blueprint
  • Pluggable lifecycle evidence (MCP or GitHub Issues) behind one port — same labeler, same Slack UX
  • Operational hardening unusual for a hackathon: structured logging, readiness probes, graceful shutdown, metrics hooks
  • Privacy-by-design: no durable store of retrieved Slack message text

What I learned

  • Agents need an authority layer separate from retrieval. Search finds noise that looks like signal; validity needs records with effective dates, supersession, and scope.
  • Deterministic boundaries make agents trustworthy. Judges (and enterprises) care more about why a verdict was assigned than about fluent prose.
  • MCP is a good fit for read-only evidence ports — lifecycle lookup as a tool, structured content only, worker orchestrates the pipeline.
  • Demo reliability is operations. Preflight scripts, seeded workspace data, and an always-on worker matter as much as the code.
  • Constraints sharpen the product. Public channels only and no retention forced a cleaner architecture than "search everything and remember it."

What's next for TruthExpiry

  • Broader claim catalog — derive schema from lifecycle records instead of a fixed Python dict
  • Production lifecycle sources — Jira, ServiceNow, or internal policy registries via the same LifecycleEvidencePort
  • Private-channel policy — explicit enterprise controls when customers need scoped search
  • Adoption metrics — track stale-answer prevention and escalation reduction in pilot workspaces
  • Tighter manifest scopes — remove unused OAuth scopes from the Slack app

Built With

  • docker
  • github-actions
  • mcp
  • mcp-(streamable-http)
  • openai
  • pydantic
  • python
  • render
  • rts
  • slack
  • slack-bolt
  • slack-rts-(assistant.search.context)
  • socket-mode
Share this project:

Updates

Submission history