Inspiration
Someone edits what a metric means. "Order Total" now excludes shipping and tax.
Nothing breaks. No pipeline fails, no test goes red, no column disappears, no alert fires. Every dashboard built on that term keeps rendering exactly as before — and quietly starts answering a different question than it did last quarter. The quarterly number moves and three teams argue about whose query is wrong.
Schema changes have guardrails. Definition changes have none. It's the failure mode with no error message, and the catalog is the only place that even knows it happened.
What it does
semantic-drift-auditor walks a glossary term's revision history, decides whether the meaning actually moved, traces everyone downstream who inherited the change, and writes the finding back into the graph.
🔴 Order Total — MEANING CHANGED
revision v5 → v6
- The total monetary value of an order, including all line items, discounts, and taxes.
+ The net monetary value of an order: the sum of all line item totals, net of
+ discounts and refunds, EXCLUDING shipping and tax.
Blast radius — applied to 21 assets, feeding 15 consumer surfaces:
3 dashboards Order Entry Dashboard (looker), Order Entry Dashboard (tableau), …
12 charts Executive Summary (powerbi), Orders By Month (tableau), …
> Every one of those 15 surfaces still renders without error. They now answer a
> different question than they did before this edit, and nothing in the stack said so.
The hard part isn't finding changes — it's not crying wolf. In the same run, a second term was also edited, but only its SQL examples changed, so it's marked CLARIFYING and stays silent.
How we built it
Python, no runtime dependencies — the audit is stdlib urllib against DataHub's APIs.
The classifier is deliberately rule-based, not an LLM call. A judge can read src/drift.py, disagree with a specific weight, and still trust the output is reproducible. An LLM verdict on a definition diff is neither auditable nor stable across runs — and this tool's whole value is that you can believe it when it says a number changed meaning.
The rule that matters most: breaking signals are counted only inside the core statement of meaning, never in the SQL examples underneath it. Rewriting the examples is a documentation change; rewriting the first sentence is a semantic one. Treating them identically produces false alarms on doc cleanups — and a drift detector that cries wolf on doc cleanups is a detector people mute.
Challenges we ran into
The API we designed around doesn't exist. Our research said to use getGlossaryTermVersions / compareGlossaryTermVersions. Those are not in the shipped OSS v1.5.0.6 GraphQL schema at all. We found out by introspecting the live schema before writing code, not after. The generic aspect version chain plus the Timeline API do the same job and are core, stable OSS surface — and the timeline actually hands back the actor and the before/after text directly, so the pivot made the evidence stronger.
Aspect version numbers are not chronological, and getting it wrong fails silently. version=0 is a copy of the current value, version=1 is the oldest, and the highest number is the newest. Diffing v0 against v1 quietly compares "now" to "the beginning of time" and still produces a plausible-looking report. We sort on systemMetadata.lastObserved instead, which is correct regardless of how GMS assigns numbers.
Glossary terms accept neither incidents nor tags. raiseIncident answers "Entity type … is not a valid destination"; addTag answers "Unknown aspect globalTags". Both attach to data assets. That turned out to be the right shape anyway — the term is fine, it's the tables inheriting the redefinition that need triage.
Judges shouldn't need 8GB of Docker. DataHub's quickstart is 7 containers. The repo ships a recorded fixture of a live seeded instance, so make demo runs the real audit code in 30 seconds with no DataHub at all — and replay output is byte-identical to live, which CI asserts on every push.
Our own demo frames caught product bugs. We rendered the terminal output and reviewed it the way a judge would see it. That found raw markdown being dumped to the TTY, DataHub's \- ingestion escapes making identical lines look different on each side of the diff, and long rewrites pushing the new definition off screen. All fixed in the product rather than cropped around.
Accomplishments that we're proud of
It knows when to stay quiet. should_alert = meaning moved AND consumers exist — a term nobody uses can be redefined freely, and a documentation cleanup never pages anyone. Re-running is idempotent: it won't stack duplicate incidents on the same dataset, so it's safe to schedule nightly.
What we learned
The strongest thing an agent can do with a catalog isn't read from it — it's put something back. A tag says "something happened here." A Document says what the definition used to be, what it is now, who changed it, and which 15 dashboards inherited it. That's the difference between an alert and inherited knowledge.
What's next for semantic-drift-auditor
Watch mode driven by DataHub Actions so drift is caught at edit time rather than on a schedule; reconciling historical figures quoted before the change; and extending the same "meaning moved" analysis to structured properties and data contracts.
Log in or sign up for Devpost to join the conversation.