Lineage Sentinel
Inspiration
Every ML incident we'd ever seen or heard about had the same shape: nothing crashed. A column got renamed three tables upstream, a freshness check quietly started failing, or live traffic drifted away from what a model was trained on, and none of it paged anyone. The failure wasn't a bug in the model. It was a broken connection somewhere in the chain of data feeding it, and that chain lived entirely in people's heads, dashboards nobody checked daily, or tribal knowledge that walked out the door when someone left the team.
DataHub already maps that entire chain (schemas, lineage, ownership, quality assertions, ML metadata) all the way from raw tables to deployed models. What was missing wasn't the information. It was something reading it proactively, on a schedule, and telling the right human before the model quietly got worse. That gap is what we set out to build.
What it does
Lineage Sentinel is an agent that walks DataHub's ML lineage graph for every production model and checks four things that traditionally fail silently:
- Schema drift: an upstream table's live schema no longer matches what a feature pipeline expects
- Quality degradation: a failing DataHub assertion on a dataset that feeds a live model, where the signal exists but never reaches the model owner
- Training/serving skew: a feature's current statistics have drifted from what the model was trained on
- Governance gaps: an upstream dataset with no registered owner or lost certification, so there's nobody to page if it breaks
When it finds something, it doesn't just log it and move on. It writes the finding back into DataHub: an incident on the model, tags on every affected asset, and a structured note with the full evidence trail, so the next engineer or agent who looks up that model inherits the answer instead of re-discovering the problem. An LLM turns the raw evidence into a plain-English "blast radius" explanation, and a second reviewer agent resolves the accountable owner (walking upstream through the lineage graph if the model itself isn't owned) and drafts the notification that actually reaches them. For the one failure mode where an automatic fix is safe (a renamed column), it even drafts a patch and opens a pull request.
How we built it
The core abstraction is a single DataHubClient interface with two implementations: a MockDataHubClient serving a hand-built, realistic lineage graph (a fraud-detection model, its feature table, two upstream tables, one deliberately broken four different ways, and one healthy control model to prove we don't cry wolf), and a RealDataHubClient that speaks the MCP Server's JSON-RPC protocol against a live DataHub instance. Every detector, the LLM triage step, and the write-back step are written against that interface, so the same code runs identically against mock data or a production DataHub deployment. Swapping one is a one-line config change.
Each failure mode is its own detector: a small, independently testable class that takes a model and the client and returns structured findings. Findings flow through an LLM triage step (Claude, with a deterministic template fallback so the pipeline never blocks on an LLM outage) into a write-back step that files the incident, tags, and note, and finally into a reviewer agent that resolves ownership and drafts the notification. We wrote unit tests for every detector against the mock graph, confirming each one fires on the broken model and stays silent on the healthy one, plus a standalone demo script that runs the entire pipeline end to end with zero external dependencies.
Challenges we ran into
Designing believable "silent" failures was harder than it sounds: it's easy to build a scenario that trivially crashes, and much harder to build one where every individual system looks fine in isolation and the problem only becomes visible by connecting the dots across lineage. We spent real time making sure each of the four scenarios in the mock graph was something that genuinely wouldn't page anyone on its own.
Deciding how much to automate was the other recurring tension. It was tempting to have the agent auto-merge fixes, but a renamed-column patch and a training-serving-skew finding call for very different levels of human judgment: we drew the line at drafting a PR for the safe case and leaving everything else as an evidence-backed recommendation for a human to act on, rather than pretending the agent should make every call.
We also had to be disciplined about keeping the write-back step honest: it would have been easy to have the triage LLM invent plausible-sounding but ungrounded detail. Keeping every explanation traceable back to a specific piece of DataHub evidence, and having a template fallback that uses the exact same evidence when the LLM isn't available, took more care than the detection logic itself.
Accomplishments that we're proud of
Getting the write-back loop actually closed (not just detection, but incidents, tags, and notes landing back in DataHub so the next person or agent inherits the context) is the piece we're most proud of, because it's the difference between a one-off script and something a team would actually keep running. We're also proud that the healthy control model stays completely silent; a detector that fires on everything is worthless, and proving it doesn't cry wolf was as important as proving it catches real problems. And the whole pipeline runs end to end, offline, with a single command and no live DataHub instance required, which matters for anyone trying to evaluate it in five minutes instead of fifty.
What we learned
The most valuable thing DataHub gives an agent isn't search: it's the connective tissue between assets that a human would otherwise have to reconstruct manually from Slack history and memory. We also came away with a much sharper sense of where LLMs add value in a pipeline like this (turning structured evidence into something a human can act on in seconds) versus where they shouldn't be trusted (deciding what's true, since that has to come from the metadata, not the model's imagination). And we learned that a good agent demo isn't the one with the most features; it's the one where every claim it makes is traceable to something real.
What's next for Lineage Sentinel
- Proper drift statistics: replace the demo's threshold heuristics with a real distributional distance metric (PSI, KL-divergence) computed from DataHub's column profiling history
- Broader auto-fix coverage: extend automatic PR drafting beyond column renames to more of the safe, mechanical fix categories
- A lightweight dashboard: a read-only view of run history and write-back activity, so teams don't need direct DataHub access to see what the agent has found
- Scheduled, always-on operation: running continuously against a live DataHub instance rather than on demand, with configurable severity thresholds per team
- Cross-model correlation: recognizing when the same upstream break is about to hit multiple models at once, and prioritizing accordingly
Log in or sign up for Devpost to join the conversation.