Drift Monitor

Inspiration

What it does

How we built it

Challenges we ran into## Inspiration

Every team loses work the same way. A requirement changes in a Slack thread ("let's do 12 months, not 30 days"), everyone agrees, and the ticket is never updated. Two weeks later a PR ships the old version, and nobody notices until a customer does. The information existed; it was spread across tools that don't know about each other.

It's getting worse: coding agents now build straight from the ticket, so they build the old version too, faster.

What it does

Drift Monitor keeps a FalkorDB context graph of a team's Slack, Discord, tickets, GitHub PRs and docs, and catches the moment they disagree.

• Drift alerts, in the thread. When a ticket owner decides something new in chat, and nothing newer replaces it, no PR ships it and nobody chose to keep the old scope, the bot replies in that thread with the graph path as evidence: Maya → message in #support-escalations → changes requirement → amends BD-101 ← PR #6 implements the older "last 30 days"
• Fix it, with the right people. The agent plans the fix from the graph (the ticket, every stale PR and doc, each assigned to its author). Only the ticket's owners can approve it, and approval writes the decision into the ticket and opens follow-up tasks.
• A company brain you can ask. Who decided this and what did it replace? Who knows the most about exports? How is this support agent connected to PR #6? If Project Alpha slips two weeks, what is hit? Every answer shows the tools it called and the graph paths it used.
• Context for coding agents (MCP). Six read-only MCP tools (current_spec, drift, who_knows, decision_history, impact, find). Asked to "implement BD-101", Claude Code without them built what the stale ticket says; with them it called current_spec, built what was actually decided in Slack, and cited the decision.
• Try it on your own repo. Paste any public GitHub repo; it builds a graph of who knows what, which PRs close which issues, open issues nobody touched, and candidate decisions in comments.
• Stays current and respects permissions. Webhooks (GitHub, tickets, Google Drive) and the Slack/Discord bots update the graph live; docs are chunked into a FalkorDB vector index and re-embedded only when they change; access control is enforced in the query, in every agent tool and at the API, so the same question gets different answers for different people.

How we built it

• Connectors for Slack, Discord, GitHub, Jira, Linear, Google Drive and docs all emit one event format, ingested with idempotent batched UNWIND … MERGE writes.
• Versions as nodes. Every ticket version and every chat change is its own Requirement, linked over time by SUPERSEDES. PRs IMPLEMENTS a specific version. That turns "did we ship what we decided?" into a short, explainable query:

MATCH (m:Message)-[:CHANGES]->(r:Requirement)-[:AMENDS]->(t:Ticket)
WHERE NOT (r)<-[:SUPERSEDES]-(:Requirement)
  AND NOT (:PullRequest)-[:IMPLEMENTS]->(r)
  AND NOT (:Decision)-[:DISMISSES]->(r)

• FalkorDB does the heavy lifting: algo.SPpaths for "how are these connected", variable-length thread paths, full-text, range and vector indexes (checked with GRAPH.EXPLAIN), GRAPH.RO_QUERY for every agent tool, and one graph per team.
• Extraction: any OpenAI-compatible LLM (we used gpt-oss-120b on Groq and qwen3:8b locally), with a rules extractor as the fallback. Every extracted change must quote its source message word for word, or it's discarded.
• Stack: Python, FastAPI, Server-Sent Events, React + TypeScript, Slack Bolt, discord.py, pytest + GitHub Actions (161 tests, ruff, mypy).

Does the graph actually matter?

We measured it. Switch off the version reasoning (SUPERSEDES/IMPLEMENTS) and drift precision falls from 1.00 to 0.50: half the alerts become false. With ticket IDs alone, both real drifts are missed, because the deciding messages never name a ticket. CI fails the build if this regresses. At scale, a 22,940-node graph found 100 of 100 planted drifts with no false alarms, and the whole-graph drift query takes 39 ms (p50).

Challenges we ran into

• A decision vs a suggestion. "Should we also do XLSX?" followed by "not this quarter" must not count. Our synthetic company plants exactly these traps.
• "Latest message wins" is wrong. Under noise, a misread status update replaced the real decision. Fixing the version chain brought both drifts back.
• Real text is much harder than synthetic. On real GitHub threads the rules extractor's flags were mostly false. A strong model was right every time it flagged something but found only 2 of 7 real decisions. We chose precision on purpose: a bot that posts in people's threads can't cry wolf.
• Looking for drift in the wild. We searched 289 real comments from FalkorDB and Excalidraw: no drift; both real scope decisions had already reached a PR. We did find one in golang/go#77441, where the accepted API lives in a comment while the issue body still proposes the old one. We found it while labelling, so it isn't a held-out result.
• FalkorDB's shortestPath is directed-only, so connection answers use algo.SPpaths in both directions and skip the feature hub, which would otherwise make every path trivially short.

What we learned

The hard part is the data model, not the LLM. Once requirement versions are nodes, "what is the current decision, who made it, and did it ship?" is a graph question with an evidence path, while plain retrieval finds the documents but can't tell which version is current.

We also learned to report the bad numbers. Every result in our repo comes with the command that produces it, including where it falls short.

What's next

Real-world pilots with teams that decide in chat, Notion and Confluence connectors, sign-in, and more agents consuming current_spec before they write code.

Accomplishments that we're proud of

What we learned

What's next for Drift Monitor

Built With

• cypher
• docker
• falkordb
• fastapi
• groq
• python
• typescript