DataLineage Guardian

Inspiration

I was inspired by a common data-operations problem: one faulty field can cause teams to block every downstream asset, even when only a few consumers are actually affected.

I built DataLineage Guardian to use DataHub column lineage for precise impact analysis and to stop automation whenever the evidence is incomplete.

What it does

DataLineage Guardian follows a Detect → Trace → Govern workflow.

It detects data-quality problems, traces affected fields through DataHub lineage, and assigns each downstream consumer one of three decisions:

$$ D(c,f,E)= \begin{cases} \text{BLOCK}, & \text{if field } f \text{ reaches consumer } c \ \text{ALLOW}, & \text{if complete evidence proves no path exists} \ \text{INCOMPLETE}, & \text{if the evidence is insufficient} \end{cases} $$

In the Healthcare demo, it detects 1,215 negative billing values, blocks mart_billing, and allows mart_demographics.

How I built it

I separated the project into two layers.

The deterministic core performs data checks, lineage traversal, Coverage Certificate generation, policy decisions, approval checks, DataHub write-back, and exact read-back.

The Agent uses the official read-only DataHub MCP Server to retrieve additional Schema, Owner, Query, and lineage context. It cannot modify decisions or mutate DataHub.

Challenges I ran into

The hardest challenge was proving that a field path does not exist. A missing path may also be caused by incomplete pagination, missing metadata, query errors, or truncated traversal.

I solved this with a hash-verified Coverage Certificate. An ALLOW decision is valid only when the certificate proves that the relevant lineage scope was queried completely.

Accomplishments that I'm proud of

• Detected 1,215 negative billing values
• Achieved 32/32 exact decisions in GuardianBench
• Passed 222/222 automated tests
• Completed a real Provider-approved DataHub write-back
• Verified every mutation through exact read-back
• Recorded 0 successful unauthorized operations

The benchmark accuracy was:

$$ \text{Guardian}=\frac{32}{32}=100\% $$

compared with:

$$ \text{TableBlockAll}=\frac{22}{32}=68.75\% $$

What I learned

I learned that an Agent should not own operational truth. Safety-critical facts and decisions should remain deterministic, while the Agent should only retrieve additional context.

I also learned that INCOMPLETE is an important safety result because it prevents unsupported automation.

What's next for DataLineage Guardian

Next, I plan to add automatic SQL-to-column-lineage generation, more database adapters, improved execution recovery, larger-scale lineage evaluation, and a human Review Case interface.

Built With

• agents
• datahub
• graphql
• mcp
• python