Lineage Sentinel

An autonomous agent that walks your ML lineage in DataHub to catch silent model-killers before they ship.

Built for Build with DataHub: The Agent Hackathon

---

The problem we went after

Most production ML failures do not come from the model. They come from the data supply chain: a feature that quietly shares lineage with the label (target leakage), a feature table that stopped refreshing but still feeds a live model (stale features), or an upstream column rename that corrupts the next training run. Model monitoring catches none of these because by the time accuracy drops, the bad model already shipped.

DataHub already has the context to catch this - lineage, freshness, ownership, ML metadata - but today an on-call engineer traces it by hand across five tools during an incident. We wanted an agent that does that traversal automatically, reasons about ML-specific risk, and writes what it learns back into the graph so the next person inherits it.

What Lineage Sentinel does

Point it at a production model. The agent walks the full path from source tables through features to model versions and deployments, then runs three checks by reasoning over metadata rather than raw data:

• Target leakage - a feature that shares a raw upstream source with the label column.
• Stale features - a feature table past its freshness SLA while feeding a live deployment.
• Schema blast radius - a proposed column change that maps through column-level lineage into a feature the model actually consumes, plus every model version and deployment downstream of it.

Each finding comes with a severity, a plain-English explanation an on-call engineer can act on, and the exact lineage path that justifies it. Then it closes the loop: one click applies a tag (leakage-suspect, stale-feature, breaking-change-watch), writes documentation, and submits a review-gated proposal so a human approves and the catalog stays trustworthy.

The key difference from shipped impact analysis: we do not just list downstream assets. We reason over the training-data-to-deployment path for ML-specific failure modes, and we make the context layer richer as we run instead of only consuming it.

Architecture

• Dashboard (Next.js App Router + Tailwind). Renders the model, an SVG lineage graph, ranked risks, the blast radius, and the write-back button. Ships with two seeded models so it looks alive on first run.
• Risk agent (/api/audit route). Traverses the lineage graph and scores risks with a deterministic engine first. If you add an Anthropic key it enriches each explanation with Claude; if that call fails it degrades gracefully back to the heuristic text.
• DataHub context. In demo mode the agent runs on a bundled synthetic ML lineage example (sources, features, model versions, deployments, dashboards, column-level lineage, freshness). Add a DataHub URL and token in Settings to point it at a live instance and enable real write-back.

Findings are framed as suspects for human review, not proofs, which is why every write-back is a governed proposal rather than a forced change.

Quick demo

1. Open the app and go to Dashboard.
2. Keep the churn_predictor model selected and click Run audit.
3. Read the ranked risks: a HIGH target-leakage flag, a stale-feature warning, and a schema change that hits live deployments. Expand any finding for the plain-English reason and the highlighted lineage path.
4. Click Write to DataHub. The tag and a governed proposal drop into the proposals panel, pending review. The loop is closed.

No API key is required for the demo. Everything runs on seeded data.

Run it

docker build -t app .
docker run -p 3000:3000 app
# open http://localhost:3000

Local development:

npm install --no-audit --no-fund --legacy-peer-deps
npm run dev

Bring your own keys

Open Settings to paste keys. They are stored in your browser localStorage only, never sent to our server, never logged, never committed.

• Anthropic (Claude) powers the plain-English explanations. Free key at https://console.anthropic.com/. Leave blank to run the deterministic demo engine.
• DataHub instance (URL + personal access token) points the agent at your own graph for live lineage and write-back. Stand one up with the DataHub quickstart: https://docs.datahub.com/docs/quickstart. Write-back needs TOOLS_IS_MUTATION_ENABLED.

Reusable DataHub Skill

skills/ml-risk-audit/ ships the same audit as an open-source DataHub Skill so any engineer can run the ML-risk workflow from Claude Code, Cursor, Codex, or Gemini CLI. It defines the inputs, the three checks, the write-back behavior, and the guardrails (SKILL.md, skill.json, checks.md).

Tech stack

• Next.js 14 (App Router) + TypeScript
• Tailwind CSS with a small custom UI kit (buttons, cards, badges, toasts)
• Custom SVG lineage graph renderer (no heavy graph dependency)
• Anthropic Claude API for reasoning, with a deterministic fallback
• Docker (standalone output) for deployment

Research we built on

• DataHub, Data Lineage for Machine Learning: Why It Matters
• DataHub docs, Lineage Impact Analysis
• The Silent Problem: Machine Learning Model Failure (arXiv 2204.10227)

Credits

Built by Aryan Choudhary for Build with DataHub: The Agent Hackathon. Licensed under Apache 2.0. Not affiliated with the DataHub project; built on its open APIs.

Built With

• css
• dockerfile
• javascript
• typescript