LineageGuard

• 

  LineageGuard — catch the upstream change before the model fails

• 

  91% of ML models degrade over time, and they decay in silence (Scientific Reports 12, 11654)

• 

  A deterministic rules engine, not a learned model — decision support, not auto-remediation

• 

  Open source at github.com/23f2001033/lineageguard

Inspiration

91% of ML models degrade over time, and they "decay in silence" — that is the finding of Temporal quality degradation in AI models (Scientific Reports 12, 11654, 2022), across 32 datasets and four industries. The cause is usually upstream data, not model drift. A column gets renamed, a type gets "fixed", a feature gets dropped — nothing errors, the pipeline stays green, and the model keeps serving predictions built on garbage. Teams find out when a dashboard looks wrong.

What it does

LineageGuard watches DataHub's lineage graph for upstream schema changes to production ML features. When one lands, it:

1. traces the blast radius through DataHub's ML lineage — dataset → MLFeature → MLModel
2. gathers evidence: recent runs, failing assertions, whether anything has already broken
3. classifies the likely failure mode with a deterministic rules engine
4. writes a structured Incident back into the graph, naming every affected model, with citations

The graph is richer after the agent runs than before. That write-back is the point.

How we built it

Python. Five stages behind one DataHubClient protocol, with two implementations: a GraphQL client for a live DataHub, and an in-memory graph for tests and offline demos. The pipeline cannot tell them apart, so the 111 tests exercise the same code the live demo does without needing Docker.

Lineage traversal uses searchAcrossLineage; write-back uses raiseIncident with a custom type. Stage 1 ships both event-driven (a DataHub Action on Entity Change Events) and polling.

The key design choice: the event is a trigger, not a payload. A schema event causes LineageGuard to re-read the schema from DataHub and diff it against its snapshot, rather than parsing a version-specific event body. It stays decoupled from any one release's event shape.

Challenges

DataHub will not let you file an incident on a model. The whole design was one incident per affected ML model, filed on the model entity where its owner would find it. It passed every test — against our own fixture. On first contact with a real GMS, raiseIncident rejected every ML URN: mlModel, mlFeature and mlFeatureTable are not valid incident destinations; only datasets and dashboards are. Incidents are now filed on the changed dataset with every affected model named in the title and payload, and the fixture was taught to reject ML URNs exactly as GMS does — a test double more permissive than the real system is worse than none.

Modelling ML lineage correctly. DataHub puts an MLFeature between a dataset and a model, so a model is two hops from the table that feeds it — a one-hop impact query finds features and misses every model. Getting that right is also what makes the tool useful: matching a changed column against MLFeature names is direct evidence the column is a named model input, not merely a column in a table something reads.

Second: keeping severity honest. With everything escalating on fan-out, every incident came out HIGH and the scale meant nothing. Severity now counts consumers, not raw nodes, and every adjustment is recorded on the incident so it can be argued with.

What we learned

An agent that never stays quiet is useless. One of the five demo scenarios exists specifically so the correct behaviour is to write nothing.

What's next

Column-level lineage where the platform provides it, which would replace name matching with exact mapping. Learned severity calibration from resolved incidents — which would make the confidence number mean what people already assume it means.

Built With

• click
• datahub
• docker
• graphql
• kafka
• python