docFreshness

Stale documentation can't hide. docFreshness flags an AI agent's context as stale the moment an upstream table it depends on drifts — and tells the agent exactly what changed and how to fix it.

Comment

How docFreshness works: DataHub's Lineage graph, Timeline API, and EntityChangeEvent stream feed a fingerprint + detector; every consumer (CLI, MCP/agent, Actions subscriber) writes the verdict back into DataHub as Structured Properties

Inspiration

At DataHub's own June 2026 town hall, co-founder John Joyce covered four reasons agent context fails in production: it's fragmented across tools, inaccessible to agents, unvalidated, and — the one this project is built for — it goes stale, because keeping it current was never an explicit process. (Source, published 2026-07-24 — paraphrased from the writeup's reporting on the talk, not a direct quotation of Joyce's words.) docFreshness is that explicit process: it binds a piece of documentation to the exact state it was written against, so a change to that state is what flags the doc — no manual re-check, no assumption that "nobody's touched it lately" means "it's still true."

AI coding agents — Claude Code, Cursor, and the rest — run on written context: CLAUDE.md files, docstrings, and the catalog descriptions that live in tools like DataHub. That context is trusted implicitly, and it silently rots.

Here's the failure we kept hitting. A doc says "revenue_summary is built from orders and customers." Six months later, someone renames a column in orders. DataHub records that change perfectly — its Timeline logs the rename, its lineage graph knows revenue_summary is downstream. But nothing connects that change back to the documentation that's now wrong. The catalog description still looks valid. The CLAUDE.md still looks valid. And every agent that reads them keeps confidently answering from a lie.

DataHub knows the schema moved. It has no concept that the docs about it are now stale. That missing layer is docFreshness.

What it does

docFreshness produces something DataHub doesn't have: a computed, queryable verdict — "this specific documentation is now untrustworthy because an upstream it depends on drifted" — along with the field-level reason and the fix.

This is exactly the shape of problem the hackathon's own Open / Wildcard track names directly — "knowledge capture" — but with the missing half most knowledge-capture tools skip: capturing a claim once is easy, knowing when it's stopped being true is the hard part. docFreshness is that second half: it binds a piece of documentation to the exact state it was written against, so a change to that state is what surfaces the doc, not a person remembering to re-check it.

The core mechanic is lineage-hop staleness. docFreshness stores a fingerprint of an entity's schema + ownership + deprecation state, plus the same signals for one hop of its lineage upstreams, bound to the moment a human or agent last confirmed the doc was accurate. When orders changes, revenue_summary's doc flips to STALE — even though the doc never mentions orders by name and was never edited. No other doc-drift tool can do this, because none of them have a live lineage graph to walk.

The unique signal — and what you can actually do with it

Once "this doc is stale, and here's why" exists as first-class DataHub metadata, five things become possible that raw change-detection can't give you:

  1. Agents check freshness before trusting a doc. Any agent on DataHub's MCP calls check_freshness before answering. A stale verdict — "stale, because orders.order_total was renamed to revenue_amount" — stops it from confidently answering off outdated context instead of hallucinating from it.

  2. The agent repairs the doc, not just flags it. The verdict ships the concrete edit ("rename column order_total → revenue_amount in docs about orders"), so an agent can apply the fix and re-verify — closing the loop. This is the "does real work" line: detection is table stakes; the applyable fix is the point. Default behavior is still propose-only, but apply-edits / the MCP apply_suggested_edits tool can act on it directly — narrowly: only renames, a literal unambiguous substitution, auto-applied; removals always stay a suggestion, because rewriting a sentence needs judgment a machine shouldn't make unsupervised. DataHub's own agent tools can already silently rewrite a description via update_description with no verification step at all — this is the gated, diagnosed version of that same write access, not a blind rewrite.

  3. A pull request that breaks a doc fails CI. docfreshness ci turns the verdict into an exit code, so a schema change that leaves downstream docs stale blocks the build until the doc is updated in the same PR. Docs stop rotting because merging is gated on them — the discipline tests give code, now for documentation.

  4. "Show me every stale doc in the catalog" becomes one query. The verdict lives in a searchable structured property, so a data-platform team gets a triage list — which docs are stale, verified against what, why — as a saved DataHub search. That actionable backlog doesn't exist in DataHub today.

  5. Real-time, low-noise flagging — live-verified, not aspirational. An event-driven DataHub Actions subscriber consumes DataHub's own EntityChangeEvent Kafka stream and stamps the flag the moment an upstream changes, with zero manual steps: we ran a real rename against a real DataHub instance with the subscriber as the only thing watching, and it correctly flagged the downstream doc within ~15 seconds, purely off the Kafka event. And it's precise: instead of "orders changed" pinging everyone on every edit, you get "these 3 docs are now wrong because of it," routable to exactly the owners who need to act.

The through-line: DataHub records changes; docFreshness produces a trust signal on documentation — and a trust signal is something you can act on automatically. An agent gates on it, a build blocks on it, a search surfaces it, a subscriber routes it. That's the difference between "the information exists somewhere in the catalog" and "the right agent or person is told, in time to fix it."

Scaling past a single doc: whole-tree checking

Real projects — especially a freelancer's — aren't one CLAUDE.md; they're a nested tree of Markdown. docfreshness check-tree <dir> walks the whole tree and resolves each file to its DataHub entity two ways: an explicit frontmatter binding (docfreshness_urn: <urn>) wins, and files without one are auto-mapped by matching known entity names in their prose (--discover pulls the candidate entities straight from DataHub). One command checks an entire docs folder and fails CI if any doc is stale. The binding is precise; auto-mapping is best-effort and honest about it — a bare name like orders can match same-named tables across platforms, which is exactly why an explicit binding is the recommended path for anything you gate on.

Proven on data we didn't design: DataHub's own nyc-taxi sample

Everything above was built and tested against our own synthetic lab. To check the mechanic wasn't secretly tuned to it, we ingested the hackathon's own official sample — nyc-taxi, ~500k real NYC Yellow Taxi trips in a genuine 3-stage pipeline (raw_trips → staging_trips → mart_daily_summary) — and ran docFreshness against it with zero code changes:

$ docfreshness status urn:li:dataset:(...,nyc_taxi.main.mart_daily_summary,PROD)
STALE  ...mart_daily_summary,PROD)
  reason: never verified

$ docfreshness verify-doc ...mart_daily_summary,PROD) --against ...staging_trips,PROD)
Verified ...mart_daily_summary,PROD) as fresh.

# rename staging_trips.total_amount -> trip_total (the real upstream table)

$ docfreshness status ...mart_daily_summary,PROD) --explain
STALE  ...mart_daily_summary,PROD)
  changed since last verified: staging_trips: column `total_amount` renamed to `trip_total`
  suggested doc edits:
    - In docs about staging_trips, rename column `total_amount` to `trip_total`.

Same lineage-hop propagation, same field-level Timeline narrative, same suggested edit — on a dataset we've never seen before, from a different platform instance, proving the mechanic is genuinely dataset-agnostic rather than tuned to our demo data.

(One useful finding along the way: nyc-taxi's own "planted freshness issues" are data-recency gaps — invisible in metadata by design, only detectable by querying row timestamps — exactly FreshnessAssertionInfo's domain, and structurally invisible to docFreshness's metadata-aspect fingerprint. Confirms the two tools solve different problems rather than overlapping.)

Every command completes a loop a native DataHub feature opens but doesn't finish

We didn't build a parallel system next to DataHub's — each piece plugs directly into a feature DataHub already ships and closes the gap in it:

DataHub feature What it gives you What docFreshness adds
Lineage graph What's connected to what Whether the documentation about that connection is still true
Timeline API That a schema changed That documentation is now stale because of it, propagated across the lineage hop
Structured Properties + Search A place to store and filter custom metadata The computed staleness signal itself — "show me every stale doc" as a real search facet
MCP Server / Agent Context Kit Read/write access to the metadata graph The computed verdict — fresh or stale — which reading raw metadata can't give an agent
Actions framework (EntityChangeEvent) A real-time event stream A subscriber that turns "orders changed" into "these 3 specific docs are now wrong," routed to exactly the owners who need to act
update_description (Agent Context Kit) An agent can already rewrite a description directly, no verification step The same write access, but gated: only the one mechanical, unambiguous edit type (renames), only after a diagnosed verdict, dry-run by default
CI / PR gates No existing equivalent for documentation health docfreshness ci blocks a merge the way a failing test would

Who it's for

  • AI agents and the teams running them — so an agent's answers about your data are grounded in context that's provably still true.
  • Freelancers and small teams juggling multiple client data projects without a dedicated data-platform team to catch drift manually — the exact gap that shows up, at enterprise scale, in DataHub's own customer stories.
  • Data-platform teams who want documentation to carry a freshness/trust state the same way data does.

How we built it

All on DataHub OSS — no Cloud-only features. Of the hackathon's five named integration paths (MCP Server, Agent Context Kit, DataHub Skills, APIs/SDK/CLI, Write-back), docFreshness uses three: its own MCP Server for the agent-facing verdict, the raw APIs/SDK/CLI (acryl-datahub) for the detection engine — an officially sanctioned path in its own right, not a fallback — and Write-back to stamp the verdict as Structured Properties DataHub already understands.

  • Fingerprint + drift detector (docfreshness.fingerprint, .detector) — hashes schema + ownership + deprecation for the entity and one lineage hop, via the SDK's batched get_entity_semityped (a check costs exactly 1 + N API calls for N upstreams).
  • Verification record as Structured Properties (docfreshness.properties) — the "verified against this state, at this time" record is stored as native DataHub structured properties, so it needs no GMS rebuild and shows up on the entity page for free. Search-filter visibility lives on a separate structuredPropertySettings aspect the CLI's properties upsert doesn't expose at all — we found this the hard way via manual UI clicks, then closed it with a one-time provisioning script (definitions/enable_search_filters.py) so a fresh install gets working search filters with no manual step.
  • Field-level explain via the Timeline API (docfreshness.timeline) — a stale verdict reads DataHub's own Timeline to name the exact field that changed and, for renames and removals, suggest the doc edit. Lineage-edge changes (a dependency added or removed) have no Timeline category at all, so those are explained separately: a direct diff of the verification record's upstream list against the live lineage graph, at zero extra API cost.
  • Event-driven subscriber via the Actions framework (docfreshness.action) — listens to EntityChangeEvents and stamps staleReason on affected downstream docs in real time. Finds them via an EXISTS structured-property filter (narrowing to entities with any docFreshness record) plus a direct per-entity read to confirm the actual match — not an exact-match search filter, which DataHub's dynamic mapping for URN-type structured properties makes unreliable (see "Challenges we ran into"). docs/DEPLOYMENT.md covers running it as a standing service (systemd/Docker), not just a one-off demo invocation — live-verified end to end, including the exact CLI invocation and a quickstart-specific schema-registry gotcha that isn't obvious from the Actions framework's own docs.
  • MCP server (docfreshness.mcp_server) — exposes the computed verdict to agents as tools (check_freshness, verify_freshness, check_catalog_doc, check_doc_text). This is the piece DataHub's own MCP server can't provide: it surfaces stored properties, but not the verdict.
  • Whole-tree doc mapping (docfreshness.docmap, check-tree) — resolves a whole directory of Markdown to DataHub entities via explicit frontmatter binding or prose-matched auto-mapping (--discover pulls candidates from DataHub), and gates CI on the entire tree, not just one file.
  • CLI + CI gate (docfreshness.cli) — check-doc, check-catalog, verify-doc, status, watch, check-tree, apply-edits, and ci, with clean exit codes (1 = stale, 2 = transport error, 3 = concurrent-verify aborted).
  • Write-back, narrowly (docfreshness.apply; apply-edits / MCP apply_suggested_edits) — auto-applies the one class of fix that's genuinely mechanical: renames, a literal word-boundary substitution with no risk of mangling prose. Removals are never auto-applied — deciding how to rewrite a sentence that mentioned a deleted column needs judgment, not a substitution — so those stay a suggestion. Dry-run by default on the CLI (--yes to write); the MCP tool never touches a file at all, it only returns transformed text for the agent's own file tools to save. Closes the loop end-to-end: propose → apply → re-verify, live-tested including the refusal case (a removal correctly stays unapplied even when asked).
  • dataJob/dataFlow coverage — documents a pipeline the same way as a table; a Spark job's doc goes stale when a dataset it reads or writes changes, via a merge-preserving write path (the SDK has no structured-property patch builder for non-dataset entities).

Everything runs against the "Widget Co." sqlite lab (customers/orders → revenue_summary), and three reproducible scripts walk the full loop live: try_it_yourself.sh (datasets), try_pipeline.sh (a Spark dataJob), and try_timeline.sh (the --explain narrative).

Why a separate MCP server, not a modified DataHub MCP server

We forked datahub-project/datahub for the PDL aspect proposal, so write access to DataHub's own MCP server wasn't actually the blocker — we could have modified it directly. We built docFreshness's MCP server standalone anyway, for reasons that have nothing to do with who's allowed to edit what:

  • Adoption. DataHub's MCP server is a generic read/write layer over the metadata graph; docFreshness computes something DataHub has no concept of at all — a staleness verdict, not raw metadata. Folding that into a modified DataHub MCP server would mean every user has to replace their standard DataHub deployment with our fork just to get it. A standalone server runs alongside any stock, unmodified DataHub instance instead.
  • Iteration speed. Forking doesn't make DataHub's own Java/Gradle/Docker build any faster to iterate on — it's the same rebuild cycle, just on a remote we control. We proved this directly while verifying the PDL proposal: a two-file model change cost a JDK install, several port conflicts, and two full environment rebuilds to verify end-to-end (see "Challenges we ran into" below). Iterating on the actual detection logic inside that stack, all week, wasn't viable on a hackathon timeline.

Either way it's a separate deployable process — forking only changes whose GitHub org it lives under, not the architecture.

Why the demo's schema change uses a direct SDK call, not a live ingestion connector

try_timeline.sh renames a column on orders by calling DataHubGraph.emit(MetadataChangeProposalWrapper(...)) on a mutated SchemaMetadataClass, rather than standing up a real sqlite file behind a live ingestion connector and altering the actual table. Worth being explicit that this is the real production mechanism, not a shortcut that skips it: every DataHub SQL ingestion connector — sqlite included — is built on the shared sql_common.py base class, which builds a SchemaMetadataClass and emits it as a MetadataChangeProposalWrapper, the identical two classes this script calls. There is no other path into DataHub's schema metadata; a connector is that same call, wrapped in a crawler. docFreshness only ever reads the stored SchemaMetadataClass aspect from DataHub — it can't and doesn't distinguish whether a connector or a direct SDK call put it there.

We chose the direct call deliberately, not only for demo convenience: a real ingestion re-crawl rebuilds the entire field list from scratch on every run, and depending on how the connector's diffing matches old fields to new ones, could register this exact change as "order_total removed, revenue_amount added" instead of a clean rename. Since removals never get an auto-suggested edit by design (see "What makes it different" and the write-back section above), that would silently break the one demo moment the rename case exists to show. Calling the SDK directly preserves field identity, so the rename is unambiguous every time — a more precise choice, not a less honest one.

What makes it different (and what it is not)

  • Not DataHub's Timeline / schema history — those record that a schema changed. docFreshness records that documentation is now stale because of it, and propagates that verdict across a lineage hop. It builds on the Timeline, it doesn't compete with it.
  • Not DataHub's FreshnessAssertionInfo (Cloud-only) — that checks data recency (did a table update on schedule), unrelated to documentation accuracy. The hackathon's own "Production ML Agents" reference architecture lists a "freshness alert" as an ML retraining trigger — same word, different concept. docFreshness's freshness is about whether documentation is still true, not whether data arrived on time.
  • Not Context Hub (Cloud) — a pre-publish human review of new AI context. docFreshness is a post-hoc, automated detector for context that's already published going stale, OSS rather than Cloud-only.
  • Not git-diff doc tools (Swimm, doc-drift, and others) — they diff the doc's own repo; they have no data-platform lineage graph, so they can't catch an upstream change the doc never mentions.

Challenges we ran into

  • Making it OSS-native without a GMS rebuild. We proved the whole loop on Structured Properties first (no rebuild), then added the first-class PDL aspect as a stretch-goal upstream PR — so v1 was demoable in a week without risking the platform.
  • Keeping the fingerprint honest. A format-pinned test guards the stored hash so a refactor can't silently invalidate every verification record.
  • A real bug the live test surfaced: re-verifying a dataset left a previously stamped staleReason lingering (the patch builder only sets). We fixed the write path to clear it, so a re-verified doc self-heals to fresh — and added a regression test.
  • A real gap the live test surfaced: the fingerprint correctly flips STALE when a dependency is added or removed from an entity's lineage, but DataHub's Timeline API has no category for lineage changes, so --explain produced nothing — a stale flag with no explanation. Verified on both our own lab and nyc-taxi, in both directions. Fixed for free: the stored verification record already carries the upstream list that was last verified against, so diffing it against the live lineage graph needs no extra API call at all.
  • Graceful degradation: when DataHub returned a server error for a dataJob's timeline mid-demo, the code fell back to the generic reason instead of crashing.
  • A real DataHub platform bug, found and root-caused while prepping the demo: Structured Properties are one of DataHub's own promoted extensibility mechanisms — and on v1.7.0, attaching one to an entity makes that entity invisible in DataHub's own main Search page. Isolated it to a single request parameter (searchFlags.includeStructuredPropertyFacets: true, which the production Search page hardcodes on every request): flip it to false with every other parameter held constant, and the identical query goes from total: 0 to the correct result. Confirmed the data was never the problem — directly querying the underlying OpenSearch index found the entity correctly indexed the whole time. Filed as datahub-project/datahub#19047 with full repro steps.
  • The same root cause bit our own code, in production logic, not just a demo UI: entities_verified_against — the function the real event-driven subscriber uses to find which docs to flag — searched for exact matches on a URN-type structured property. DataHub dynamically maps that field as OpenSearch text, not keyword, so the exact-match search silently returned zero results, meaning a deployed subscriber would have missed every real staleness event without ever logging an error. Fixed by switching to an EXISTS filter (unaffected by the mapping issue) to get the candidate set, then confirming each candidate's actual verified-against list with a direct per-entity read instead of trusting search to filter precisely. Caught by actually running the production code path end to end against live DataHub, not just unit tests against a mock.
  • Getting the real subscriber running for the first time surfaced two more gaps between the Actions framework's own docs and reality: the CLI command is datahub-actions actions run, not datahub-actions run (the latter fails immediately with "No such command"); and the pipeline's default SCHEMA_REGISTRY_URL assumes a standalone Confluent registry on port 8081, which a datahub docker quickstart deployment doesn't run — GMS serves its own internal registry instead, at <gms-host>/schema-registry/api/. Worse, the wrong URL fails silently: the pipeline logs "is now running" either way and only a wrong URL never processes a single event, no error anywhere. Confirmed by testing both the correct and incorrect URL back to back against the same live instance.

Accomplishments we're proud of

  • The lineage-hop verdict works end-to-end and live: rename a column on an upstream, and a downstream doc that never mentions it flips to STALE — visible natively on the DataHub entity page.
  • Detect and repair: the verdict hands an agent the exact edit, and for the one mechanically safe case — a rename — can apply it directly and re-verify, with a live-proven refusal boundary (a real column removal correctly stays unapplied even when explicitly requested).
  • Explains not just schema drift but lineage-edge drift (a dependency added or removed) — a signal DataHub's own Timeline API has no category for at all, closed at zero extra API cost.
  • Four integration surfaces (MCP, CLI, CI gate, event-driven subscriber) over one detection core, 87 tests, ruff-clean.
  • The event-driven subscriber isn't just code-complete — it's verified live, twice, end to end: a real schema rename, consumed off a real Kafka topic, correctly flagging the right downstream doc with zero manual intervention, in both runs.
  • Verified dataset-agnostic against DataHub's own nyc-taxi sample, not just our lab — zero code changes needed, every scenario (schema, ownership, lineage-edge, documentation) re-confirmed identically on both platforms.

What we learned

The valuable thing wasn't detecting a schema change — DataHub already does that. It was realizing the gap was a missing binding: nothing tied a piece of documentation to the state it was written against, so nothing could compute "this doc is now stale." Adding that one primitive unlocked agents, CI, search, and real-time alerting all at once.

What's next

  • The first-class docFreshnessInfo PDL aspect merged upstream into DataHub.
  • Column-level lineage (flag only docs about the specific changed column).
  • A fourth fingerprint signal — captured query/view-definition text — to catch silent business-logic drift where schema and lineage stay identical.
  • Coverage for dashboards and ML models.

Built with

Python · DataHub OSS (Structured Properties, Timeline API, Actions framework, Metadata SDK) · Model Context Protocol (MCP) · Typer · GitHub Actions · sqlite

Built With

  • actions-framework
  • datahub-oss
  • github-actions
  • metadata-sdk
  • model-context-protocol-(mcp)
  • python
  • sqlite
  • structured-properties
  • timeline-api
  • typer
Share this project:

Updates