Provenance

Trust Report
The catalog summary
The lineage walk
Write-back verification
Three Grade tags
Browser

Inspiration

DataHub shows how data assets are connected, but lineage alone does not answer the most important question:

Can this asset actually be trusted?

A dashboard may depend on datasets across Power BI, Looker, Snowflake, dbt, and S3. Some upstream assets may contain PII, have no assigned owner, lack documentation, or carry unhealthy incidents.

Finding these problems manually means opening every asset in the lineage chain and comparing its governance metadata one node at a time.

We built Provenance to automate this investigation, produce an evidence-bound trust grade, and write the result back into DataHub as searchable catalog knowledge.

What it does

Provenance is a lineage-governance agent for DataHub.

It performs the following workflow:

Resolves a requested asset through DataHub search.
Walks its complete upstream lineage.
Retrieves governance metadata from every node.
Evaluates six weighted governance checks.
Produces a score from 0–100 and a letter grade.
Attaches the exact node and metadata fact behind every deduction.
Writes the grade back into DataHub as a Provenance-Grade-X tag.
Reads the graph again to verify that the write succeeded.

The result does not remain inside a terminal report. It becomes durable catalog knowledge that another analyst, workflow, or agent can search and reuse without running the audit again.

How we built it

Provenance follows a deterministic five-stage pipeline.

1. Resolve

The requested asset is resolved by name through the DataHub MCP search tool.

Provenance uses the canonical URN returned by DataHub instead of manually constructing one.

2. Walk

The agent recursively traverses the complete upstream lineage and deduplicates previously visited nodes.

The traversal is deterministic. A language model does not decide which lineage node should be visited next.

3. Extract

Every node is batch-fetched, and only governance-relevant metadata is retained:

Ownership and stewardship roles
Glossary terms
Sensitive classifications
Tags
Documentation presence
Domains
Incident health
Lineage relationships

This converts large entity responses into compact and auditable evidence records.

4. Score

Six weighted checks calculate a reproducible score and grade.

The same DataHub metadata produces the same result. Every deduction preserves the evidence that caused it.

5. Write back and verify

The final grade is emitted to DataHub as a Provenance-Grade-X tag with a description explaining its evidence basis.

Provenance then reads the graph aspect back and reports:

VERIFIED IN GRAPH: True

A successful request alone is not treated as proof that the metadata was persisted.

The scoring system

Provenance evaluates six governance checks:

Unowned sensitive data upstream: Detects PII, SOC2, or GDPR-classified nodes without an assigned owner.
Ownership coverage: Measures how much of the upstream chain lacks ownership.
Missing upstream lineage: Penalizes assets whose provenance cannot be verified.
Documentation coverage: Identifies undocumented nodes in the chain.
Upstream incident health: Detects unhealthy statuses anywhere upstream.
Missing data stewardship: Identifies sensitive root assets without an assigned data steward.

Every finding includes the affected node URN and the specific DataHub metadata fact that produced it.

Provenance does not generate unexplained trust scores. Every deduction can be checked directly against DataHub.

Technologies used

Python 3.12
DataHub MCP Server
DataHub Python SDK
DataHub Core v1.5.0.6
DataHubGraph
DataHub Search API
DataHub Lineage API
DataHub Entity API
DataHub Metadata APIs

The audit uses metadata already stored in DataHub, including lineage, ownership, stewardship roles, glossary classifications, tags, documentation, domains, and incident health.

Provenance does not fabricate missing freshness, ownership, or health information.

Results

We ran Provenance across eight assets in the DataHub showcase catalog.

Asset	Platform	Grade	Score	Upstream nodes	Findings
Customer Analytics Measures	Power BI	F	23/100	13	4
Geographic Measures	Power BI	F	23/100	13	4
order_details	Looker	F	23/100	13	4
ORDER_DETAILS	Snowflake	D	45/100	23	3
orders	Snowflake	D	51/100	1	4
CUSTOMERS	Snowflake	C	70/100	0	2
PRODUCTS	Snowflake	C	70/100	0	2
WAREHOUSES	Snowflake	C	70/100	0	2

A representative Grade F audit found:

Eleven upstream nodes carried sensitive classifications without an assigned owner.
Only 15% of the upstream nodes had an assigned owner.
Ten of thirteen upstream nodes lacked documentation.
The sensitive root asset had no assigned data steward.

Every deduction identified the exact node and metadata fact that caused it.

The key insight

Trust decreased as the lineage moved closer to the decision layer.

Raw Snowflake source tables scored Grade C at 70/100.

Snowflake assets with upstream dependencies fell to Grade D, including ORDER_DETAILS at 45/100.

The Looker and Power BI assets scored Grade F at 23/100.

The downstream analytics layer inherited governance gaps from its upstream dependencies while also carrying governance requirements of its own.

This catalog-wide pattern was invisible until Provenance audited the complete lineage chain.

Challenges we ran into

Reliable DataHub write-back

During development, DatahubRestEmitter could accept a metadata change proposal without reliably persisting it through the asynchronous endpoint.

We moved write-back to the synchronous DataHubGraph.emit() path and added mandatory read-after-write verification.

Self-hosted infrastructure limits

Running eight catalog searches rapidly on an 8GB host exhausted the OpenSearch container.

Provenance now paces queries between batch audits to keep the environment stable.

Canonical asset identifiers

URNs copied from browser URLs were percent-encoded and did not always match the identifiers stored in the graph.

We solved this by resolving every asset through DataHub search and using the canonical URN returned by DataHub.

Large entity responses

A single DataHub entity response can contain thousands of characters of metadata.

Instead of passing complete responses through the workflow, Provenance extracts only the governance-relevant fields immediately after retrieval.

Missing freshness metadata

The showcase catalog did not contain reliable freshness timestamps.

Instead of inventing a staleness signal, we constrained the audit to governance evidence that genuinely existed.

Accomplishments that we are proud of

Built deterministic multi-hop lineage traversal.
Created a reproducible evidence-bound scoring system.
Preserved node-level evidence for every deduction.
Identified sensitive upstream assets without accountable owners.
Wrote trust grades back into DataHub.
Verified every write directly in the graph.
Made trust grades searchable across the catalog.
Audited eight assets to expose a catalog-wide governance pattern.
Published the source code, setup instructions, and sample outputs under the Apache 2.0 license.

What we learned

Reading metadata is useful, but contributing a verified result back to the graph is much more powerful.

Once a trust grade becomes a first-class DataHub tag, it can be searched, filtered, reused, and inherited by other people and agents.

The audit becomes part of the catalog instead of disappearing when the terminal closes.

We also learned that missing metadata must be handled honestly. An asset with no recorded lineage or ownership should not receive a clean bill of health simply because there is less evidence to inspect.

Silence is not evidence of safety.