Silent-Drift Sentinel

An on-call AI agent that catches a silently degrading ML model, traces it through DataHub lineage to the upstream column that broke it, and writes the cause back so the next agent inherits it.

Comment

Inspiration

The person who detects a model degradation is almost never the person who caused it. A data engineer fixes a normalization bug in an upstream table, the model was trained on the old values, accuracy slides for two weeks, and nobody connects the pipeline change to the model, because the two live in different tools owned by different teams. Roughly 80% of production ML failures trace to data and pipeline issues, not model weights. That gap, coordination without authority, is what we set out to close, with DataHub as the bridge and the place the agent leaves knowledge for the next agent.

What it does

The Silent-Drift Sentinel is an on-call AI agent for the ML engineer who owns a silently degrading production model. In one human-gated run it:

  1. Detects the degradation from a label-free performance signal (NannyML CBPE), so it fires before ground-truth labels arrive, not just from raw input drift.
  2. Walks DataHub's ML lineage to the exact upstream column that changed.
  3. Identifies the owning team from catalog ownership.
  4. Generates the data-quality guardrail that would have caught it (a dbt test, a Great Expectations expectation, a SQL guard) from the diagnosed change type.
  5. Writes a durable drift_causation object and the generated proposed_fix back onto the model, raises an incident on the upstream dataset, and pings the owning team in Slack, so the next engineer or agent inherits the diagnosis.

Then it closes the loop: run it again and the agent reads the finding it already wrote, recognizes the known cause, and stops. No re-diagnosis, no duplicate incident. That is the thesis made literal: an agent leaving knowledge on the catalog that the next agent inherits.

It also proves it is not a toy. A three-way scenario toggle (harmful null regression, a structurally different default-value regression, and a benign unit rescale) shows the detector correctly staying silent on the benign shift and alarming only on the harmful ones. Most drift tools cry wolf; this one does not, and we measured it.

How we built it

  • ML core: a calibrated LightGBM model on the real UCI Online Shoppers dataset, honest temporal split, isotonic calibration verified with ECE (0.087 to 0.041) and Brier. A two-layer detector: label-free CBPE plus per-feature drift (KS, Chi-squared) with Benjamini-Hochberg FDR correction, a PCA reconstruction check, and data-quality fingerprinting. An evaluation harness scores the detector over five labeled scenarios: alarm precision and recall are both 1.0, root-cause localization is 1.0, and change-type accuracy is 1.0, all measured, not asserted.
  • Agent: a LangGraph state machine (Detect, Recall, Traverse, Root-Cause, Identify Owner, Propose Fix, Write-Back) with durable Postgres checkpointing and Langfuse tracing. Root-cause has two modes: a fast single synthesis, or an opt-in agentic loop where Claude itself decides which catalog reads to make and each read streams to the UI live. Claude writes only the prose narrative (with a Gemini failover via LiteLLM); every catalog write is deterministic code behind a write-ahead log, gated by a human-in-the-loop interrupt, so a read never mutates the catalog.
  • DataHub reads and writes: reads are deterministic DataHub Python SDK aspect reads (model to feature to source table to owner). The write-back is split because DataHub's incident metamodel does not allow incidents on ML models: a drift_causation structured property, a proposed_fix structured property, a drift-degraded tag, and the RCA on the model, plus an incident on the upstream dataset via the GraphQL raiseIncident mutation. A read-back endpoint re-fetches the property from DataHub as proof it landed, not the agent's claim.
  • Surfaces: a Next.js 16 dashboard streaming the run over SSE, with a React Flow lineage graph, ECharts drift charts, a model card, the detector-eval tile, and the live write-back reveal. Plus a native Expo mobile app (iOS and Android) that approves the write-back from a phone. Deployed on Vercel; the agent and a self-hosted DataHub Core (metadata auth on, least-privilege service-account PAT) run on an always-on VM behind Caddy TLS.

Use of DataHub, and contributing back

We go beyond reading metadata. The agent writes a new context class (drift_causation) and a generated guardrail (proposed_fix) back onto the model, raises a real incident on the upstream table, and on a re-run reads its own finding back to short-circuit. Two upstream contributions:

  • The datahub-ml-drift-rca skill to the DataHub Skills registry (datahub-project/datahub-skills PR #29), so any agent can perform this workflow.
  • A docs issue on the write-path authorization gotcha (structured-property writes need GraphQL, not the Rest.li /aspects emit, under a least-privilege policy) and the incidents-cannot-target-mlModel constraint (datahub#18273).

Originality (framed honestly)

Drift detection and owner lookup are commodities; we concede them. The novel parts are automated model-lineage root-cause and the write-back of a new context class onto the model entity in an open-source catalog, closed by reading it back on the next run. Prior art stops short in specific ways: Monte Carlo and Bigeye root-cause data incidents, not model degradation; Arize and Fiddler work in feature space, not catalog lineage; Atlan and OpenMetadata hold the metadata but leave the trace to a human. Root cause here is lineage-guided correlation plus data-quality evidence, stated as such, not proven causation.

Accomplishments

Everything runs live on a real hosted DataHub with no mock backend, verified end to end (a judge clicking the link runs the real agent). A measured eval, not a claim. An opt-in agentic loop you can watch Claude drive. A close-the-loop recall. Metadata-aware code-gen. A CI matrix (Python + Web + Mobile) that gates every commit. Web and mobile on the same agent.

What we learned

DataHub's write authorization differs by path: a least-privilege token can write structured properties via GraphQL but not through the Rest.li /aspects emit. Incidents attach to datasets and jobs, not to ML models, which shaped the split write-back. The async graph search index returns empty lineage on a freshly emitted catalog, so deterministic aspect reads are the load-bearing traversal. Each of these became either an upstream contribution or a documented design choice.

Challenges

Keeping the language model out of the write path while still letting it investigate. We solved it with a two-mode root-cause (fast synthesis or a watchable agentic loop) whose only output is prose, and a deterministic, idempotent write path behind a write-ahead log and a human approval gate, so the live demo cannot wobble.

What's next

Continuous scheduled monitoring against a live production window (the demo runs on-demand), multi-model coverage, and turning the generated guardrails into opened pull requests on the upstream dbt project.

Try it

Built With

Share this project:

Updates