PolicyLineage

From policy text to provable action across every dependent data and AI asset.

Comment

Inspiration

A new policy can be published today, but it does not name the dataset, model, pipeline, dashboard, or owner who must act tomorrow. Regulatory-intelligence products understand documents but not an organization's real technical dependencies. Data catalogs understand dependencies but not the operative meaning of a new rule. PolicyLineage was built to close that gap.

The initial scenario comes from advanced-research collaboration and economic-security work, where a plausible exception, a missing approval, and one hidden downstream export can completely change the answer. We wanted a system that made uncertainty and counter-evidence visible instead of producing an opaque “risk score.”

What it does

PolicyLineage is an evidence-first regulatory compiler for DataHub. It:

  1. converts its supported synthetic policy pack into typed conditions, path conditions, exceptions, and source-located evidence;
  2. searches DataHub and traverses multi-hop lineage to find the real affected datasets, jobs, dashboards, models, owners, and external surfaces;
  3. keeps a heuristic impact-priority index separate from an evidence-coverage index;
  4. produces a deterministic ALLOW, REVIEW, or BLOCK decision and a SHA-256 evidence bundle;
  5. proposes DataHub tags, structured properties, and a related decision document;
  6. requires a human to approve that exact digest;
  7. re-reads DataHub before writeback and refuses stale approval if any policy or catalog fact changed;
  8. writes the approved decision back so the next human or agent inherits verified context; and
  9. preserves the decision, analysis context, attributable approval/writeback state, and supersession link in a searchable Decision Registry.

The demo's synthetic Operation Kintsugi scenario first finds controlled quantum-control data reaching an external collaboration job through three lineage hops. It blocks the change, shows five affected assets and the possible fundamental-research exception, then simulates replacement with approved public benchmark data. Recompilation changes the decision to ALLOW and supersedes the old evidence bundle.

How we built it

  • DataHub 1.6.x Python SDK seeds deterministic datasets, schemas, owners, tags, pipelines, dashboards, an ML model, lineage, and PolicyLineage structured properties.
  • The official DataHub MCP Server provides search, get_lineage, and get_entities for context, then add_tags, add_structured_properties, and save_document after approval.
  • FastAPI + Pydantic expose typed policy, evidence, catalog, decision, and mutation contracts.
  • A deterministic Python compiler evaluates entity predicates and required reachable-path predicates, traverses cycles safely, and fails closed when the graph is incomplete.
  • A SQLite audit ledger reserves each digest once, records approvals and write results, links superseded decisions, and prevents blind mutation replay.
  • The authenticated Control Plane adds isolated workspaces, hashed and revocable API keys, four least-privilege roles, attributable audit events, a durable revalidation queue, and PostgreSQL persistence without weakening the approval boundary.
  • Read-only registry endpoints expose newest-first decision summaries and complete audit records without creating a second source of truth; DataHub remains the organization record.
  • Live DataHub routes require bearer authentication; the approver identity comes from trusted server configuration rather than the request body.
  • The build-free HTML/CSS/JavaScript UI keeps the public fallback reliable and exposes policy mapping, context calls, graph coverage, evidence, exact digest, and downloadable JSON.

What makes it different

PolicyLineage is not a policy chatbot and not another lineage viewer. Its product unit is a versioned, falsifiable decision bundle connecting three graphs:

  • the policy graph of conditions and exceptions;
  • DataHub's organization graph of assets, governance, ownership, and lineage;
  • the decision graph of evidence, approval, writeback, and supersession.

The external-transfer condition is a real lineage predicate: controlled data alone does not block. It blocks only when a reachable DataHub asset exposes the transfer surface.

Challenges we ran into

The hardest engineering boundary was freshness. Binding approval only to a prior digest was insufficient because an asset could change between analysis and mutation. We made semantic hashes stable across retrieval timestamps, stored the original search and policy context, and added a full pre-write revalidation. Changed context returns 409 and performs zero mutations.

We also treated truncated lineage as a first-class state. “No edges returned” is not the same as “no dependencies exist,” so an incomplete graph can never silently produce ALLOW.

Accomplishments

  • End-to-end BLOCK → remediation → ALLOW → new approval flow in a real browser.
  • Exact official MCP read and mutation contracts with partial-graph handling.
  • Public clean-room DataHub 1.6.0 rehearsal: eight resolved assets, seven lineage edges, six governed entities, all three approved mutation tools successful, and post-write tag plus structured-property verification.
  • Approval-bound, idempotent writeback with stale-context rejection.
  • Visible Decision Registry covering immutable bundles, approval state, and supersession.
  • Workspace Control Plane with viewer, analyst, approver, and admin roles, PostgreSQL-backed isolation, attributable audit events, and durable revalidation jobs.
  • 14-case expert-designed synthetic regression benchmark with 1.00 verdict accuracy, affected-asset precision/recall, exception accuracy, replay stability, approval-gate rate, incomplete-graph safe routing, and identity-attribute invariance.
  • 62 executable tests pass with 83% Python coverage at this audited checkpoint.
  • A validated reusable auditable-data-change Agent Skill submitted as upstream DataHub Skills PR #45.

These benchmark results validate the published synthetic fixtures and compiler boundary; they are not claims of legal accuracy. The two displayed indexes are transparent heuristics for review prioritization and evidence coverage, not calibrated probabilities.

What we learned

Context quality must be part of the decision, not an invisible implementation detail. Agents also need durable memory in the catalog: a correct answer that disappears into a chat transcript cannot govern the next change. Finally, exposing counter-evidence makes human review faster because reviewers can see what would falsify the result.

What's next

  • Connect the durable revalidation queue to DataHub metadata-change events and production schedulers.
  • Add enterprise OIDC/SAML, signed policy-pack distribution, and retention-policy controls.
  • Add expert-reviewed packs for AI training-data permissions, cross-border transfer, licensing, and research funding while retaining the deterministic enforcement boundary.

Built With

Share this project:

Updates