Causal Drift Sentinel

• 

  Intervention delta separates genuine root cause from confounded nodes

• 

  Full upstream chain: datasets → features → model → deployment, causal path highlighted

• 

  ML lineage graph fetched from DataHub — root cause isolated to raw_user_profiles

• 

  Structured diagnosis with evidence trail and suggested fixes, written back to DataHub

Inspiration

Three weeks. That's how long it took a fraud team to realize their model had gone blind.

Not because their monitoring failed. It fired alerts every day. But every alert said the same useless thing: "Model performance degraded." No cause. No direction. Just an angry Slack message and engineers staring at dashboards at 2am.

The root cause turned out to be a single column — account_age_days — quietly truncated by a schema migration upstream. One field. Three weeks. Millions in undetected fraud.

I built Causal Drift Sentinel because that story is not rare. It happens at every company running ML in production. The monitoring exists. The lineage exists. Nobody connected them.

What it does

Causal Drift Sentinel is an AI agent that watches ML models in production and — when something breaks — tells you exactly what caused it, not just that it happened.

It reads the full ML lineage from DataHub via the MCP Server: raw datasets, feature tables, models, deployments — the entire upstream chain. When drift is detected, it walks backward through that lineage DAG and runs intervention-style causal checks on every node. It separates the genuine root cause from confounded nodes that merely look suspicious.

The result: a structured diagnosis in seconds, written back into DataHub so the next engineer — or the next agent — inherits the answer instead of starting from scratch.

How I built it

• DataHub MCP Server — the backbone. Reads full upstream lineage, writes incident annotations back via add_tags + update_description
• FastAPI — five-stage pipeline: ingest → detect → trace → reason → write-back
• React + Vite — lineage graph visualization, drift timeline, and root-cause report in one interface
• Groq (LLaMA 3) — grounded LLM layer with a hard contract: it cannot name a cause the causal algorithm didn't already isolate
• KS-test + PSI + Intervention Delta — the causal isolation engine. Not correlation. Actual intervention scoring.

The hard part nobody talks about

Detecting drift is easy. Every monitoring tool does it.

The hard part is that a feature table downstream of a drifting dataset will show identical drift statistics to the dataset itself. A naive agent flags both. An on-call engineer gets two suspects and no answer.

The intervention delta metric solves this: it measures how much downstream drift collapses when you intervene on a given node. The genuine root cause collapses 78% of drift. The confounded feature node collapses almost none. One number. Clear answer.

This only works because DataHub gave the agent the full lineage context to walk through. Without it, there is no graph to intervene on.

What's next

Real-time streaming detection, Slack alerting, and native DataHub incident tracking as the MCP API matures. The write-back path is already there — it's just waiting for the incident tool to ship.

Built With

• apache
• datahub
• docker
• fastapi
• github-actions
• groq
• llama-3
• mcp-server
• pytest
• python
• react
• tailwind-css
• typescript
• vite