ColdLineage

DataHub can tell you a table is cold. It cannot tell you that half a table is cold — and it cannot move a single byte. ColdLineage does both and writes the receipt back into DataHub.

Comment

The problem, stated precisely

Every data platform team has a stalled "what can we drop?" project. It stalls because nobody can prove a deletion is safe, so nothing gets deleted, and the warehouse bill compounds.

DataHub already solves the detection half of this, and solves it well. Metadata Tests can continuously find tables in the bottom quartile of usage. Impact Analysis shows what depends on them. Deprecation and soft-delete mark them. DPG Media cut Snowflake spend 25% doing exactly that. None of that is what this project claims to add.

Two things remain, and they are the two that actually block the work:

  1. DataHub's model is dataset- and column-level. It can say "this table is unused". It has no way to say "rows before 2023 are cold while the last 90 days are hot". It has partition vocabulary — PartitionSpec carries a timePartition — but that describes what a profiling run measured, not a lifecycle state you can set, search or act on. So the enormous middle case — a heavily-queried table whose first four years nobody has read in years — is invisible to dataset-level tiering. That case is most of the savings, and it is the one that terrifies people, because archiving a live table feels reckless.

  2. Nothing in DataHub moves data. Soft delete, DataHubGC, deprecation and Metadata Tests all operate strictly on metadata. DataHubGC deletes stale metadata rows by age; it has never moved a byte of warehouse data.

ColdLineage occupies exactly that gap.

How it decides

The question "is this date range safe to archive?" reduces to one measurable thing:

How far back does each downstream consumer actually read?

DataHub knows that a dashboard depends on a table. It does not know that the dashboard only ever reads the trailing twelve months. But it does hold the dashboard's real SQL, as Query entities.

So ColdLineage reads that SQL out of DataHub and parses it with sqlglot, resolving the lower bound each consumer places on the subject's date column into a concrete date:

Predicate in the consumer's real SQL Resolved history window
event_date BETWEEN DATE '2024-01-01' AND CURRENT_DATE 2024-01-01
event_date >= CURRENT_DATE - INTERVAL '90 days' 2026-05-08
date_trunc('month', event_date) >= date_trunc('month', CURRENT_DATE - INTERVAL '18 months') 2025-02-01
event_date > (CURRENT_TIMESTAMP - INTERVAL '24 months')::date 2024-08-06
WHERE performing_lab IS NOT NULL unbounded — blocks every cutoff

A cutoff is safe iff it is no later than the earliest window across all active consumers.

That last row is the whole point. It has a WHERE clause, so a "does this query filter?" heuristic passes it. It has no date bound, so it reads every row ever written.

Everything unproven blocks

The parser is deliberately pessimistic, because getting this backwards deletes data someone was still reading. All of these resolve to unbounded, which refuses the archive:

  • no date predicate at all
  • a boolean OR where any branch is unconstrained
  • NOT over a date predicate
  • a bound we can parse but cannot resolve to a date
  • the subject read inside a CTE whose outer query filters it
  • SQL we fail to parse
  • lineage we could not read at all — absence of evidence is not evidence of safety

Under AND the effective bound is the latest of the branch bounds; under OR, the earliest, and unbounded is contagious. 20 tests in backend/tests/test_window_extraction.py pin this down.

How DataHub is used

Not as a decoration on the side of a local database. Every decision input is read from DataHub at request time, and every value carries a provenance tag that is rendered in the UI, so a missing input shows up as a visible gap rather than a plausible-looking number.

Read

GraphQL against GMS — every document validated against a live v1.7.0 schema:

Signal Source
Downstream consumers searchAcrossLineage
The SQL each one runs listQueries → queryProperties.statement
Usage: last query, 30d count, distinct users datasetUsageStatistics
Retention floor, legal hold, business criticality Structured properties io.coldlineage.policy.*
Schema, date column, owners, domain, tags, terms schemaMetadata, ownership, domain, tags

Write

Four contributions to the source entity after a verified archive, plus a catalog entity for the archive itself. Nine of nine landed in the verified run below:

Contribution Mutation Why
6 typed archive properties upsertStructuredProperties Machine-readable facts, validated and scoped
Deprecation note + decommissionTime updateDeprecation The warning a human sees on the entity
Manifest link addLink Where the bytes went, clickable
cold-tier-archived tag addTag Makes the archived set searchable

And the archive becomes its own dataset — s3.coldlineage-archive/<table>/<cutoff> — because the bytes are a governed asset in their own right: PHI, a schema, a retention obligation, and somebody will eventually need to find them without already knowing which source table to look behind. Five aspects, emitted as MetadataChangeProposals to POST /aspects?action=ingestProposal (backend/app/datahub/frozen.py):

Aspect What it carries
datasetProperties Cutoff, rows, bytes, part count, sha256, manifest URI, source URN
schemaMetadata The source's columns and types, carried over (12 for patient_encounters)
subTypes Cold Tier Archive
upstreamLineage COPY from the source — the same rows moved, not derived
globalTags cold-tier-frozen-copy, so "where are the archives?" is one search

A sixth, erModelRelationship, retargets the source's declared foreign keys onto the frozen copy — built only from constraints the database actually enforces, never from two columns that happen to share a name, because a guess written into a catalog becomes a fact somebody else plans against. The demo estate declares no foreign keys, so that path is a documented no-op here and is covered by unit tests instead.

Deliberately not done: writing the datasetProperties aspect wholesale. That aspect holds other writers' custom properties and a whole-aspect PUT silently destroys them. Structured properties are typed, validated, entity_types-scoped, and survive other writers. The definitions are committed in backend/app/datahub/properties.yaml.

Ingest uses the first-party postgres connector plus the acryl-datahub Python SDK, so the demo estate is real catalog content with real URNs — not hand-built fixtures.

Where the rows sit

Each table's rows split three ways — archivable, held by policy (provably unread but inside the retention window), and in use (a consumer can still reach it). Only the middle band answers to configuration. Sweep retentionYears on patient_encounters and watch what does not change:

Retention Floor Archivable Held by policy In use
14 years 2012-08-08 0.0% 60.6% 39.4% (433,161)
4 years 2022-08-08 41.7% 18.9% 39.4% (433,161)
2 years 2024-08-08 60.6% 0.0% 39.4% (433,161)
4 months 2026-04-08 60.6% 0.0% 39.4% (433,161)

Between rows two and three the binding constraint flips from policy to evidence, and after that the knob does nothing at all. Retention is a floor, not a permission slip — the band that refuses to move is fixed by WHERE e.event_date BETWEEN DATE '2024-01-01' AND CURRENT_DATE, read out of DataHub and parsed. Drive it with scripts/set_policy.py, which writes the policy to DataHub: retention belongs to a governance owner, not to the tool that benefits from relaxing it.

Agent surface — two, sharing one executor

Surface Description
agent/ A standalone agent that reads DataHub through the official MCP Server — search, get_lineage, get_dataset_queries, get_entities, list_schema_fields, get_lineage_paths_between — and acts through six constrained operations, blocking on a human before any delete. Runs on Claude or GPT.
skills/assess-data-temperature/ The same decision procedure as a loadable DataHub Skill, for anyone already in a skills runtime (Claude Code, Cursor, …), driving the datahub CLI.

Neither holds database credentials. See Trust boundary.

The agent being provider-agnostic is the same argument as the trust boundary, made twice: if the safety of the system depended on which model you plugged in, it was never safe. One JSON Schema definition becomes both Anthropic and OpenAI tool formats, one system prompt serves both, and the approval gate lives in the provider-neutral executor where no driver can route around it — with tests asserting the two providers are handed identical tools and that the executor imports no model SDK. The OpenAI driver targets the bare Responses API, so OPENAI_BASE_URL also points it at Azure, vLLM, Ollama, or a self-hosted model.

Two commands

Requires Docker Desktop (≥8 GB to Docker) and Python 3.11+.

# 1. DataHub itself — a real external catalog, exactly as it would be in your environment
pip install 'acryl-datahub[datahub-rest]'
datahub docker quickstart

# 2. ColdLineage
git clone https://github.com/Abhinav0905/ColdLineage.git
cd ColdLineage
make demo          # brings up Postgres + MinIO + API + UI, seeds 3.65M rows, ingests into DataHub
Service URL
ColdLineage UI http://localhost:3100
API docs http://localhost:8000/docs
DataHub http://localhost:9002
MinIO console http://localhost:9001 (minioadmin / minioadmin)

If a port is already taken (8080 and 3000 are common), override and re-run:

DATAHUB_GMS_URL=http://host.docker.internal:8090 docker compose up -d

No DataHub? Run the recorded catalog. DATAHUB_MODE=replay serves verbatim GMS responses committed in examples/cassettes/. The UI labels the mode and the recording timestamp, and every signal is tagged cassette:recorded rather than datahub:*. There is deliberately no third mode that invents context.

The demo estate

Five synthetic healthcare tables, 3,650,000 rows / 566 MB measured (pg_total_relation_size, never declared). Four different date-column names on purpose — a parser that hardcodes one looks like it works and is wrong.

Table What it isolates
patient_encounters The hero. Temperature 81.3 HOT — genuinely in active use — yet 516,088 rows (46.9%) sit before 2023 and every consumer reads no earlier than 2024-01-01. Archivable.
lab_results The killer. Temperature 10.8 COLD — 0 queries, 0 users in 30 days. Every dataset-level tool archives it tomorrow. Blocked at every possible cutoff — one HIPAA extract does an unbounded scan.
claims_history ACTIVE legal hold (MDL-2291) as a DataHub structured property. Range analysis approves; policy vetoes.
care_events_live Genuinely hot; the 2-year retention floor lands before the table starts.
billing_ledger Consumers clear a 2022 cutoff; the 7-year retention floor does not. Same table, different cutoff, different answer.

Those two rows are the entire argument:

patient_encounters   81.3 HOT    -> archivable      (46.9% of it is provably unread)
lab_results          10.8 COLD   -> blocked         (at every cutoff, forever)

Dataset-level temperature gets both of them exactly backwards. Only reading each consumer's actual SQL separates them.

Verified run

Against DataHub OSS v1.7.0, reproducible with make examples:

patient_encounters   1,100,000 rows / 178 MB / event_date 2019-01-01 .. 2026-08-05
  temperature 81.3 HOT, archive_eligible: true

cutoff sweep
  2022-01-01  SAFE_TO_ARCHIVE           +730d
  2023-06-01  SAFE_TO_ARCHIVE           +214d
  2023-11-15  ARCHIVE_WITH_REHYDRATION   +47d
  2024-03-01  DO_NOT_ARCHIVE             -60d   blocked by Quarterly Compliance Dashboard

EXECUTE cutoff=2023-01-01
  516,088 rows -> 11 Parquet parts -> s3://coldlineage-archive/...
  read-back digest match: true | rows 516,088/516,088 | schema match: true
  -> source deleted only after verification.  1,100,000 -> 583,912
  DataHub writeback: 9/9 operations ok   (4 on the source, 5 aspects on the frozen archive)

RESTORE  516,088 rows rehydrated, SHA-256 verified

Trust boundary

flowchart LR
  subgraph reasoning["Reasoning — no data-plane credentials"]
    S[agent/ via MCP<br/>or the DataHub Skill]
  end
  subgraph context["Context — DataHub"]
    DH[(GMS)]
  end
  subgraph executor["Executor — constrained, 4 operations"]
    P[plan] --> H{human approval}
    H -->|plan hash| X[execute]
    X --> V[verify read-back]
    V -->|pass| D[delete hot rows]
    V -->|fail| A[abort, source intact]
    R[restore]
  end
  S -->|reads| DH
  S -->|plan / simulate| P
  X --> M[(Parquet + manifest)]
  X -->|provenance| DH
  M --> R

The reasoning layer never receives DDL/DML authority — no database credentials, no object-store client, no ability to issue SQL. It gets read-only MCP tools plus six constrained operations, and that tool list is the guarantee: it holds even if the model is wrong or the prompt is attacked — or, indeed, if it is a different model entirely. A human stands between plan and execute. Approval is a plan hash binding dataset + cutoff + row count + verdict — if live state drifted since the plan was shown, execution is refused rather than proceeding against different data.

This is deliberately the opposite of handing a model a database connection and a careful prompt.

The ordering inside execute is the safety argument:

  1. stream rows out in chunks → multi-part Parquet
  2. upload parts, then the manifest
  3. download the parts back from object storage
  4. recompute SHA-256 on the retrieved bytes and compare
  5. re-read the Parquet, assert row count and column set
  6. only then delete — in one transaction
  7. re-count; roll back on any mismatch

Step 3 is the one that matters. Hashing the buffer you are about to upload proves nothing about what landed.

What is honest about the numbers

At demo scale the storage saving is about one cent a month. The verified run moved 666,839 rows out of Postgres and landed 28.7 MB of Parquet, and no amount of framing makes that a business case.

The API reports the measured figure alongside the unit rates it actually used — hot $0.115/GB-month, cold $0.004/GB-month, a $113.66/TB-month delta — plus the archived fraction, and does not round the measured one up into looking impressive. The hot rate is deliberately warehouse-attached storage (managed-Postgres SSD), not S3 Standard at $0.023/GB-month: the bytes being freed sit in a database, not in a bucket. Priced against S3 Standard the same delta would be $19.46/TB-month, and quoting that would flatter the number by pretending the hot side was already object storage.

What transfers is the fraction, not the dollars: 60.6% of a live, actively-queried table turned out to be provably unread. Applied to a real estate that ratio is the entire argument, and it is measured, not modelled.

Also stated plainly: rows are exact; per-range byte figures are estimates — the table's measured physical size apportioned by row share, because Postgres does not track per-range size. Every estimate is labelled as one in the API response.

Limitations

  • Postgres only. The executor moves Postgres → Parquet. Other platforms appear in lineage as consumers but are not archive candidates; listing a Snowflake table as archivable when the only executor is Postgres would be a claim the product cannot honour.
  • Restore of ~500k rows takes ~75s. Correct, not fast; it round-trips through pandas.
  • The estate is synthetic. Every ingested entity is stamped coldlineage.synthetic=true. Row counts, byte sizes, column types and date ranges are measured from the live database.
  • Multi-hop consumers inherit the earliest bound of their upstreams rather than an exact mediated edge. This can over-protect — block a cutoff that was in fact safe — never the reverse.
  • searchAcrossLineage lags the graph index by ~1 minute after ingestion.

Layout

Path Purpose
backend/app/services/window.py The differentiator — SQL → history window
backend/app/services/simulation.py Cutoff → verdict
backend/app/services/archive.py The constrained executor
backend/app/datahub/ GraphQL reads, writeback, cassettes, property definitions
agent/ The MCP-driven agent — the tool list is the security model
skills/assess-data-temperature/ The loadable DataHub Skill
scripts/ Estate, consumers + their real SQL, DataHub ingestion
examples/ Artifacts from a real run — readable without running anything
CONTRIBUTING-UPSTREAM.md Proposed upstream contributions to DataHub

DataHub technologies used

The Devpost picker only offered a few options, so: DataHub MCP Server + Other.

Here is the full surface, all of it against DataHub OSS v1.7.0.

1. MCP Server (mcp-server-datahub 0.6.0)

The agent reads the catalog through the official MCP Server over stdio, with mutation tools off, so it cannot write to DataHub on its own:

  • search
  • get_lineage
  • get_dataset_queries
  • get_entities
  • list_schema_fields
  • get_lineage_paths_between

2. GraphQL API — every decision input, read at request time

Capability Purpose
searchAcrossEntities Discover the estate
searchAcrossLineage Downstream consumers, multi-hop with degree
listQueries Each consumer's real SQL (queryProperties.statement.value)
datasetUsageStatistics 30-day query count, distinct users, last query
structuredProperties Retention floor, legal hold, business criticality
schemaMetadata Columns, types, the date column
ownership / domain Owners and domain
globalTags / glossaryTerms Classification
deprecation Existing deprecation state

listQueries is the one the whole project turns on. DataHub already stores each consumer's statement verbatim; we parse it with sqlglot to derive how far back that consumer actually reads. Nothing else exposes that.

3. GraphQL API — writing the receipt back

Mutation What ColdLineage writes
upsertStructuredProperties 6 typed properties under io.coldlineage.archive.*
updateDeprecation Note carrying the cutoff and the restore path
addLink institutionalMemory link to the archive manifest
createTag + addTag cold-tier-archived, so the set is searchable

Safety note: ColdLineage deliberately never writes the datasetProperties aspect wholesale—it holds other writers' customProperties, and a whole-aspect PUT destroys them.

4. OpenAPI / REST — the archive as its own entity

Endpoint: POST /aspects?action=ingestProposal

MetadataChangeProposals register the frozen archive as a first-class dataset:

Aspect What it carries
datasetProperties Cutoff, rows, bytes, sha256, manifest, source URN
schemaMetadata The source's 12 columns and types, carried over
subTypes "Cold Tier Archive"
upstreamLineage COPY from the source—same rows moved, not derived
globalTags cold-tier-frozen-copy
erModelRelationshipProperties The source's declared FKs, retargeted

5. Structured Properties — typed, not customProperties

Definitions are committed in backend/app/datahub/properties.yaml.

Property Purpose
io.coldlineage.policy.retentionYears Minimum age before cold (fractional)
io.coldlineage.policy.legalHold + legalHoldMatter Legal-hold policy and matter
io.coldlineage.policy.businessCriticality Business criticality
io.coldlineage.archive.* 6 properties written after an archive

Typed, validated, entity_types-scoped, and they survive other writers.

6. Python SDK (acryl-datahub 1.7.0)

  • First-party postgres ingestion connector
  • MetadataChangeProposalWrapper + schema_classes + DataHubRestEmitter

Used to ingest the demo estate: five datasets, seven consumers, Query entities holding real SQL, structured-property definitions, and lineage. Real catalog content with real URNs, not hand-built fixtures.

7. Skills

Skill Purpose
skills/assess-data-temperature/ The same decision procedure as a loadable DataHub Skill, driving the datahub CLI

8. Entity types touched

dataset · dashboard · chart · dataJob · dataFlow · mlModel · query · tag · structuredProperty · erModelRelationship

9. Replay mode, for reviewers

33 verbatim GMS responses recorded from a live v1.7.0 are committed in examples/cassettes/. DATAHUB_MODE=replay serves them, so the app runs without standing up DataHub—and every value is tagged cassette:recorded with its capture timestamp, so recorded data is never passed off as live.

What we did not use

Assertions / data quality, Forms, the Actions framework, and Metadata Tests (Cloud-only). Named here because overclaiming is worse than a short list.

Upstream contributions

While building this we found that DataHub's own skills/datahub-enrich/references/mutation-reference.md documents upsertStructuredProperties incorrectly — the example fails schema validation three ways (structuredPropertyInputs should be structuredPropertyInputParams, values takes [PropertyValueInput!]! not bare strings, and the mutation returns StructuredProperties! so it needs a selection set). Proof and a draft PR are in CONTRIBUTING-UPSTREAM.md, alongside a proposal to contribute this skill upstream — datahub-skills currently has no skill covering cost, storage, tiering, retention, archival or lifecycle.

Built With

  • datahub
  • fastapi
  • graphql
  • mcp-server-datahub
  • next.js
  • openai
  • pyarrow
  • python
  • render
  • sqlalchemy
  • sqlglot
Share this project:

Updates