Inspiration
Sensitive fields are tagged at the source — any location that incurs data movement.Sensitive fields are tagged at the source, wherever it is that data is moved. A glossary term on a user record (PII) tag on users.email But that data never sits still. It is copied into Snowflake and is transformed by dbt and rolled. Up to PowerBI, Tableau and Looker dashboards. The classification remains in place These fans out to all the readers of the source table while the actual sensitive data fans out across the stack, unlabeled. Governance doesn't end at the data; it doesn't end at the source.
I continually came back to the same ungainly image: an order_history table that is of a structure I have never used.
is definitely classified by PII and exists in the catalog with no classification,
it. If metadata has not been implemented, then there is no way to enforce a policy. As the amount of stack increased,
As it is driven by agents and automated pipelines, that disparity is growing. I wanted
governance should be transitive — to go with the data wherever it went.
What it does
If lineage and existing tag/terms exists in DataHub, then it is used by TagFlow. It is to be hoped that downstream assets have an upstream sensitive classification. and records and returns those classifications back into the catalog — with provenance, so All changes are recorded. Determine the requirements of a downstream asset that already runs on the creek. It is also called a different label, it ***flags the conflict and won't be overwritte
DataHub's tags can be propagated along lineage. TagFlow isn't a reimplementation It's the pointy-headed brainy guy at the top here — it's the governance safety layer and agent. We're detecting and refusing safely, so its a dry-run workflow time! → apply workflow It writes and a json run report which you can diff.
How I built it
In a clean read → decide → write loop in the DataHub Python SDK:
Read a lineage from the server, including N hops (default = 20) lineage.Read offspring from the server (max_hops=N defaults at 20, try greater values to go deeper into the lineage tree).
No hand rolled graph traversal.
Read the classification of some GlobalTags, using the low-level method graph.get_aspect(urn, GlobalTagsClass / - ซึ่งเป็นGlobalTags Classificationที่สำคัญ
GlossaryTermsClass).
Ignore the writeback event for datas if it is emitted by - - - - **Write back** viaMetadataChangeProposalWrapper` aspect emits, so datas
dbt models, charts and dashboards are labeled using the same consistent syntax.
path.
Small and accurate model. Ignore the edges, consider a directed graph $G = (V, E)$. where by the directed edge, u $\rightarrow$ v, is the flow of data from u toward v. The set of a source downstream it. $s$ in $k$ hops is
$$D_k(s) = {\, v \in V : 1 \le d(s, v) \le k \,},$$
with $d$ the shortest directed path length. For an extra sensitive classification The propagation targets are living on the source set ( $S_c$ ).
$$T_k(c) = \bigcup_{s \in S_c} D_k(s),$$
and TagFlow writes $c$ to a target $v \in T_k(c)$ only if $v$ is an element of target $T_k(c) \rightsetminus c$. There is no other conflicting sensitive label of the same type.
commutative operation.Propagates like a commutative operation, writes are idempotent. closure: a second pass is a fixpoint:
$$P(P(x)) = P(x),$$
That's why, when it's run again, it writes nothing, because it was already written. And since the "newly tagged $v$",
become sensitive sources for the following run, the classified frontier compounds
; determine value of image url when trying to auto-discover image url toward that fixpoint: on a live showcase-ecommerce catalog.
Absolute sizes of the sensitive sources changed over runs from $|S_c| = 2 to |S_c| = 21, for the re-run
correctly writing $0$.
On that live DataHub, one PII_Data tag for one Order Entry source,
reached 34 classifications in 69 scanned downstream assets, and
0 write failures, up to 5 hops out. Safety is incorporated during the dry run: dry-run by
The tale of a conflict "default" will either succeed or fail one at a time.default/conflict refusal/per-entity fault isolation ("33 succeeded, 1 failed")
It does not abort the run (never) and it has scoping: --source / --limit N. The decision logic
It has a pytest suite running against the fake DataHub that is stored in memory, so it
Does not require a connection to follow the live signal.
I encountered several impediments.I faced a couple of hurdles.
- The subtle, self inflicted bug, I added to avoid re-reading the same entity.
A cache which provides write coherence per run. Added an automated conflict detection: To detect break automatically:
The first two labels (e.g. PII, GDPR) from the same source were labeled as sibling.
They didn't seem to agree with each other when written out. The fix
Was meant to take a
conflict_basissnapshot of target's existing labels. only after conflicts against it are detected before cache. Now the regression is covered by a pytest case. As mentioned - identity, not names. Early idem was comparing tags by name rather than identity. A real propagation may be hidden with the same-named-but-distinct label. Moving the states that are more likely to be present, using the word "already present?" check to get it aligned with the way the writer states it. actually dedupes. - Infrastructure. A low-spec laptop that enabled DataHub's 'quickstart' to run in a
Not locally, GitHub Codespace — and it was the aspect path write-back that was the one.
Also, I didn't want to make an educated guess of the piece, so it remains in
tags.py, and checked it to make sure that it works correctly. Compare it to the True graph. While this is not unexpected, I was told by being honest about novelty that DataHub ships with this already. lineage-aware tag propagation. In place of overclaiming, I repositioned TagF: Top on top of that, the safety-audit part. Particularly sharp story and true.
What I learned
DataHub's model and aspect of reading and writing; how to make governance a data class path To build transitive closure of a lineage graph, not a per-table afterthou why idempotency needs to be proved at the URN level; that the honest The framing era of a project is commonly the solid one. The whole thing is releas Licensed under the Apache-2.0 license, and using the DataHub's open source SDK.
What's next
An MCP server wrapper that allows any MCP client (such as Claude) to call propagate from a conversation a packaged DataHub Skill to the propagation runbook, and a Accommodate conflicts in the flow triage (particularly when having a financial reason to accommodate them).
Log in or sign up for Devpost to join the conversation.