Context Drift Agent

The problem

Schema changes don't break documentation loudly. They break it silently.

A field goes from FLOAT to VARCHAR. The description still says "stored as a numeric value in the account currency." The glossary term still says NumericMetric — "a field that stores a numeric financial measurement." Nothing errors. No pipeline fails. No alert fires.

But that documentation is now false, and everything downstream is consuming a contract that no longer exists: dashboards, ML features, and — increasingly — AI agents that read metadata to decide how to use a table.

Metadata doesn't have a correctness monitor. Data quality tools watch the data. Nobody watches whether the description is still true.

What it does

Context Drift Agent watches DataHub metadata and tells you the moment it stops being true.

Schema change detected (polling)
        │
        ▼
Context Retrieval
  ├── GraphQL              → description, glossary terms, custom properties
  └── DataHub MCP Server   → downstream lineage, owners, domain
        │
        ▼
LLM Judge — "Is the existing documentation still accurate?"
        │
        ▼
Writeback to DataHub customProperties
  context_stale = true
  context_confidence = 0.95
  context_drift_reason = "The field credit_limit changed from FLOAT to
    VARCHAR(16777216). The existing description explicitly states it is
    'stored as a numeric value', which is now inaccurate. The glossary term
    NumericMetric is also no longer appropriate for a string-typed field."

The signal lands in DataHub itself, so every person and every agent reading that metadata inherits it. No new dashboard to check, no separate system to integrate.

It judges context — it does not diff schemas

This is the part we care most about.

A schema-diff tool reports every change. That produces alert fatigue, and alert fatigue produces ignored alerts. So we tested the inverse case:

Schema change	Verdict	Confidence	Reasoning
`FLOAT` → `VARCHAR`	stale	0.95	The description claims a numeric value; it no longer is
`VARCHAR` → `FLOAT`	not stale	0.85	Going back to numeric improves alignment with the docs

The second row is the whole point. The field changed, but the documentation became more accurate, not less — so there is nothing to flag. An agent that cried wolf on every type change would be worthless in a catalog with thousands of tables.

Detection to writeback takes 7–13 seconds end to end, including the LLM call and MCP enrichment.

MCP-native in both directions

The agent reads from DataHub's own MCP Server (mcp-server-datahub) to enrich the judge's prompt with downstream lineage, owners, and domain — so the verdict can weigh blast radius, not just the field in isolation.

The agent also exposes itself as an MCP server, so any other agent can ask it directly:

Tool	Purpose
`check_drift(urn)`	Run an on-demand drift evaluation
`get_drift_status(urn)`	Read the last recorded verdict
`list_monitored_datasets()`	List what is being watched

Drop it into a Claude Desktop config and you can literally ask "check for drift on the customers table."

How we built it

Python 3.11+, DataHub OSS, uv. Roughly 20 small single-responsibility modules. The pipeline stages are independently testable and each logs clearly, which mattered more than we expected when debugging LLM behaviour.

Three architecture decisions we documented and would defend:

customProperties over Structured Properties. Typed Structured Properties are the "correct" DataHub primitive, but the OSS API surface is still stabilising. customProperties are string key-value pairs that render in the UI today, on any DataHub OSS install, with no schema registration step. We chose the thing that works for users now and documented the migration path.

Polling over the Actions Framework. The Actions Framework means Kafka, event handling, and deployment complexity. Polling a GraphQL endpoint every N seconds is a loop and an HTTP call. For an MVP watching tens of datasets, the event-driven version buys nothing and costs a lot.

Structured output, always. Every LLM call is forced tool use with a schema requiring context_stale, context_confidence, and context_drift_reason. There is no free-text parsing anywhere in the judge path. An explanation the operator cannot read is a verdict they cannot trust.

Two features go beyond single-pass detection:

Adversarial evaluation (behind USE_ADVERSARIAL_JUDGE). A prosecutor argues the context is stale, a defender argues it is still valid, and the verdicts are synthesised. When they disagree, confidence is capped and both arguments surface in the reason — so operators see the uncertainty instead of a false-confident number.

Synthetic context validation. Before any schema change, the agent can generate three questions a downstream AI agent might ask about a dataset, then test whether the existing description and glossary terms can answer them. It writes context_answerable and context_qa_confidence back to DataHub. This answers a different question: not "did the context break?" but "was it ever good enough?"

Challenges we ran into

The MCP Python SDK renamed its high-level server class. FastMCP moved to MCPServer under mcp.server.mcpserver. Our MCP server module could not even be imported — which meant the context-drift-mcp entry point our README documented was completely broken.

The uncomfortable part: all 45 tests passed with that module dead, because not one of them imported it. A test suite that green-lights an unimportable module is measuring the wrong thing. We wrote a smoke test that imports it, asserts the three documented tools are registered, and checks the console script in pyproject.toml still points at the right function. That test would have caught it on day one.

Two mocks were lying to us. tests/test_client.py set block.type = "message" on a mocked Anthropic content block — but "message" is the response type, not a content-block type; the real SDK uses "text". Another test set no type at all. Both passed only because the code under test indexed content[0] blindly.

That blind index was a real latent bug: if the API had returned a thinking block first, the agent would have crashed in production. Fixing the type guard broke the tests, which is exactly what should happen when mocks don't match reality.

Type checking found what tests could not. pyright flagged 17 errors that the passing suite never surfaced, including that same content-block access and an MCP content union where only TextContent carries .text. Our CI now runs four gates: ruff check, ruff format --check, pyright, and pytest — 51 tests, all green.

Getting the LLM to not fire. Early prompts flagged almost every type change. The fix was not a bigger model; it was asking the right question. We stopped asking "did the schema change?" and started asking "is this specific sentence still true given this specific change?" — and gave the judge the description and glossary text verbatim, not a summary.

What we learned

Metadata is a contract, and nobody was testing it. We started thinking of this as a documentation-hygiene tool. It is closer to a type checker for the gap between what a schema is and what its documentation claims.

Not firing is a feature. The single most valuable behaviour is the VARCHAR → FLOAT case returning not stale. Precision is what makes an alert worth reading.

Explainability is not optional for metadata. A boolean context_stale flag with no reason is unactionable — the operator still has to reconstruct what broke. The human-readable reason field is what makes the verdict something you can act on in one read.

A passing test suite is not coverage. Our most instructive bug was invisible to 45 green tests and obvious to a type checker.

What's next

Proactive validation loop — run synthetic context validation automatically on every drift event, writing context_stale and context_answerable in one pass
Multi-dataset dashboard — aggregate drift and sufficiency signals across hundreds of tables for data quality teams
Structured Properties migration — move from string customProperties to typed properties once the OSS API stabilises
Field-level glossary terms — attach verdicts to the specific column that drifted, not just the dataset

Positioning

DataHub Cloud announced Context Platform in May 2026 — a paid, private-beta product that generates new context from query logs and SME review.

This project does something different and complementary: