Verity — Organizational Knowledge Integrity Engine

Verity lives in Slack. A developer changes rateLimiter.js. Documentation goes stale. Six months later, the same outage repeats — unless Verity's Neo4j graph and GitHub MCP catch the drift first.

Comment

Inspiration

Every engineering team has felt this: someone changes a file, the code compiles, the tests pass, CI is green — and everything looks fine. But the documentation describing that file didn't change. The runbook didn't change. The Architecture Decision Record didn't change. Six months later, a different engineer touches the same file and repeats an incident that already happened once, because nothing in the toolchain ever connected the code to the knowledge around it.

GitHub doesn't know what your docs say. Your docs don't know what Jira says. Slack doesn't know any of it changed. We wanted to build something that closes that gap automatically, inside the tool engineers are already in every day — Slack — instead of adding one more dashboard nobody checks.

What it does

Verity is a Slack-native knowledge integrity engine. Every alert, every question, every action happens inside Slack — no new dashboard, no context switch. It watches a repository, and whenever code changes, it:

  1. Pulls the current, live file state through the GitHub MCP server — never a stale webhook payload.
  2. Updates a Neo4j knowledge graph connecting that code to documentation, ADRs, runbooks, incidents, Jira tickets, owning teams, and Slack discussions.
  3. Recursively traverses that graph using Cypher to calculate a blast radius — everything downstream that might now be out of date. This is visualized in a live, clickable, severity-color-coded graph page, not just a text list.
  4. Retrieves supporting evidence for each affected artifact from the graph and from a custom Slack Conversation Retrieval Layer.
  5. Passes that evidence to an LLM reasoning layer (Groq / Llama 3.3), whose only job is to summarize and cite what was actually retrieved — never to invent facts.
  6. Posts a Drift Card directly in Slack, with the changed file, impacted docs, responsible team, related incidents, and a live link to the blast-radius graph.
  7. Can draft a documentation fix from cited sources, which a human reviews and explicitly Approves, Edits, or Discards before anything is restored.

Beyond that reactive path, Verity also works proactively and conversationally:

  • Predictive drift on PR open. The same blast-radius pipeline can run against any diff, pre-merge or post — so a reviewer could catch drift at review time instead of after deploy.
  • A conversational agent, right in Slack. Ask @Verity who owns this service? or use /verity, and it answers in natural language, citing exactly which graph nodes ([G]), Slack messages ([S]), or GitHub commits and PRs ([H]) it drew the answer from — the same evidence-only discipline as the Drift Cards, just conversational.

If Verity can't find enough evidence to answer a question, it says so directly instead of guessing.

How we built it

  • Backend: TypeScript, Node.js, and Express handle the GitHub webhooks, Slack endpoints, and the core drift-processing pipeline.
  • Knowledge graph: Neo4j stores every artifact as a node (Code, Document, ADR, Jira, Runbook, Service, Owner, Incident, Slack Thread) connected by real relationships (DOCUMENTS, IMPLEMENTS, DEPENDS_ON, OWNS, GENERATED_FROM, MENTIONS). Blast-radius analysis runs as a recursive Cypher shortestPath traversal.
  • GitHub integration: the official GitHub MCP server gives Verity a standardized way to pull live file contents, commit metadata, and pull request context on demand, instead of trusting whatever the webhook payload happened to contain.
  • Slack integration: built on the Slack Bolt SDK — Events API, Slash Commands, Block Kit, and threaded replies — so the entire experience lives inside Slack with zero extra dashboard.
  • Evidence retrieval: a custom retrieval layer that combines Neo4j graph results with Slack's search.messages API for past conversation evidence, with a seed-data fallback for reliable demoing.
  • Reasoning layer: Groq (Llama 3.3) sits at the very end of the pipeline. It only summarizes and cites evidence that's already been retrieved — it never originates facts, and it says so explicitly when evidence is missing.
  • Deployment: Docker Compose for local Neo4j, with the app itself running on Node/Express.

Challenges we ran into

  • Ensuring MCP added real value beyond webhooks. We enforced that Verity always re-fetches live file state through MCP, since webhook payloads can be stale or incomplete by the time they're processed.
  • Avoiding hallucinated organizational knowledge. It's easy to have an LLM confidently invent an owner or an incident that doesn't exist. We built the pipeline so the reasoning layer is evidence-only — if the graph and Slack retrieval don't surface something, Verity says it doesn't know, rather than guessing.
  • Graph traversal vs. keyword/vector search. Early on we considered a simpler RAG-style approach (embed docs, do similarity search). We moved to graph traversal instead because similarity search can't answer "what else breaks" or "who owns this" — those are relationship questions, not similarity questions.
  • Keeping the Slack experience genuinely native, not a thin wrapper — buttons to open the graph, review evidence, or approve/discard a drafted fix, all without leaving the channel.

Accomplishments that we're proud of

  • A real, working evidence-grounded pipeline — not a mockup — where every Drift Card and every AI-drafted fix is traceable back to specific graph nodes and specific retrieved evidence.
  • A conversational agent that answers direct Slack questions with multi-source citations ([G] graph nodes, [S] Slack messages, [H] GitHub history) — not a black-box chatbot, a cited one.
  • A human-in-the-loop trust layer: nothing gets "fixed" automatically. Every AI-drafted documentation update requires explicit human Approve/Edit/Discard before it's restored.
  • Recursive blast-radius analysis that actually reflects real dependency chains (DEPENDS_ON, OWNS, GENERATED_FROM, MENTIONS), not just files that look similar to each other.
  • An honest, precise account of what we built — we know exactly which parts are Slack's own platform (Bolt SDK, Events API, Block Kit) and which parts are our own custom layers (evidence retrieval, reasoning), and we didn't blur that line.

What we learned

That "who owns this" is a graph-distance question, not a similarity question. Vector search finds docs that look like the code. Cypher traversal finds the ADR that decided the code, the incident that broke it, and the Slack thread where the team agreed to change it. That's the difference between search and understanding.

What's next for Verity

  • Native Notion and Confluence sync, so documentation updates can flow both directions.
  • Automatic Jira issue creation for detected drift.
  • Multi-repository enterprise graphs and cross-team dependency intelligence.
  • CI/CD-stage impact analysis, so blast radius is visible before a PR even merges.

Built With

Share this project:

Updates