Assay

Assay prices every asset in your DataHub catalog and tells you what it costs when one breaks.

Comment

Inspiration

Every data platform team has the same conversation, and it never resolves.

Someone asks whether a table can be dropped. Someone else opens the lineage view, sees a web of arrows, and says "probably, but I'm not sure." Nobody signs the ticket. The table stays, and it keeps being rebuilt every night, forever.

The reason that conversation stalls is not missing lineage. It's that lineage tells you what is connected, not whether anyone would care. "Can we drop this" is an economic question wearing a graph-shaped costume, and no catalog answers it — DataHub included. It knows the table exists, who owns it, what it feeds. It has no idea what it's worth.

DataHub itself sells this problem. Their product page pitches finding tables with "zero reads in the past 30, 60, or 90 days — so you can delete them and reclaim storage," and their ROI page quotes "up to 25% reduction in Snowflake storage costs" and customers saying "$250,000 to $300,000 per year." Every primitive that argument needs is already in the open-source metadata model. Nothing assembles them into a number. The ROI page is a spreadsheet a salesperson fills in by hand.

So we built the spreadsheet — computed from your actual catalog, written back into it, and re-derived by CI on every push.

What it does

Assay attaches two numbers to every asset in a DataHub catalog — what it costs, and what it costs you when it breaks — then writes both back into the graph so every other human and agent inherits them.

On a 1,561-asset estate:
Annual spend $2.6M
Recoverable $996k/yr
Value at risk $59k/day
Never-read columns 3,023 ($274k/yr)

And the split nobody surfaces. Of the $2.0M Assay derives, storage is $16k. Rebuild compute is $1.84M — 92%. Storing a dead table is cheap. Recomputing it every night is not, and no catalog attributes that to a table today.

That number is derived, not configured. operation is a timeseries aspect: every rebuild appends a row, and Snowflake, BigQuery, dbt and Airflow ingestion all emit it. Assay takes the median interval between rows. Point it at a catalog carrying no Assay metadata whatsoever and it still produces real money:

operations emitted derived cadence rebuild cost, 2 TiB table
hourly 24.00/day $126,144/yr
nightly 1.00/day $5,256/yr
weekly 0.14/day $751/yr

Median rather than mean, because one backfill would drag an average into nonsense. Rows under a minute apart are one rebuild reported in parts. A single operation yields no cadence — it records that a rebuild happened, not how often — because "unknown" is a different fact from "never".

Every verdict carries its counter-evidence. Assay's most common recommendation on an expensive asset is not delete this:

OVERSERVED   recover $78k/yr   acme.support_raw.agent_shifts_v7
  + rebuilt 24x/day but read 2.53x/day
  + rebuild cost $87k/yr
  − asset IS in use — reduce cadence, do not delete

UNPRICEABLE is a first-class outcome. Assay refuses to judge assets it lacks evidence for, because recommending deletion of a table you have no usage data for is how a tool gets banned from an organisation.

How we built it

Deterministic, all the way down. No model call decides a verdict. The same graph and policy produce the same verdicts byte-for-byte, forever — which is what makes them reviewable, diffable in CI, and safe to write into a shared catalog. An LLM may narrate a verdict; it may not change one. The whole economic core runs offline with no API key.

Reads DataHub's context graph, deeply. Eight entity types and seven aspects: datasetProfile, datasetUsageStatistics (including per-column fieldCounts[], which is how we find columns nobody has ever read), operation history, chart/dashboard usage, upstreamLineage, ownership/domains/terms/tags, and DataHub's own cost aspect — the one place cost is first-class in the OSS model.

Writes the economics back as five structured properties namespaced io.assay.*, searchable as facets in DataHub's own UI. This is the point of the project: once they're in the graph, every other agent inherits them. An incident bot that used to say "orders_daily is stale" can say "orders_daily is stale, and it carries $47k/day of downstream exposure" — without knowing Assay exists. Verified end-to-end by reading back from a separate client, because a writer that reports its own success proves nothing.

Four ways in: a CLI, a React/WebGL lineage canvas rendering 1,561 nodes at 60fps, an MCP server so agents can ask rather than re-derive, and a DataHub Actions plugin so the economics stay current unattended.

Challenges we ran into

  • The live read never worked, and looked like it did. _hydrate's GraphQL merged Dataset.properties.name (String!) with Chart.properties.name (String) into one response key — a static validation error, so GMS rejected the whole document — and a per-chunk except Exception: continue swallowed it. Every live scan returned 0 lineage edges, 0 owners, 0 tags and printed a plausible estate anyway. Every offline test was green throughout. edges 0 → 1,684.

  • We had shipped the same bug elsewhere. Four bare except: pass blocks wrapped the four aspects that are the cost model. One GMS timeout turned a 4 TiB hourly-rebuilt table into $0 with no counter and no warning. Failures are now counted per aspect and carried into the report.

  • The headline only worked on catalogs we'd seeded. Rebuild cadence — the input to 92% of the number — came from a custom property our own seeder wrote. On a stranger's DataHub it would be absent and every rebuild cost would be $0. That's what forced the derivation from operation history.

  • DataHub Cloud aspects are a trap. usageFeatures / storageFeatures are in no OSS entity registry. Load the showcase datapack into DataHub Core and it prints Filtered 248 incompatible MCPs — then reports success.

  • Structured properties index silently wrong. A value is only searchable if its definition already carried a searchConfiguration when the value was written. Adding one later doesn't retroactively index, and nothing says so: the write succeeds, the read succeeds, only the filter comes back empty.

Accomplishments that we're proud of

We measured whether the economics change the decision, instead of asserting it. assay ablate gives four choosers the same 20-asset deprecation budget and scores them against a ground truth that deliberately never references Assay's own verdict logic — a test enforces that by AST walk.

Arm Recovered/yr Harmful picks
Rank by size — what every catalog encourages $169,467 17 / 20
Zero reads in the warehouse — what a FinOps tool sees $189,885 16 / 20
Nothing-downstream — lineage without economics $173,235 16 / 20
Assay $233,598 5 / 20

The third row is the one we're proudest of, because it answers the hardest question about this project: Snowflake's ACCOUNT_USAGE is ground truth for cost — why is this a catalog tool? So we built that baseline and handed it the query counts at full fidelity, better data than Assay has. It still proposes deleting live assets 16 times out of 20 — because a table with zero direct reads feeding a dashboard forty people open every morning is invisible from inside the warehouse.

And the harm count understates it. Size-ranking's 17 wrong picks touch $23,916/day of downstream exposure. Assay's 5 touch $0. Its mistakes are about assets nobody depends on.

Across 40 runs on 10 independently generated estates, Assay wins every one and is never more harmful than the baseline in any of them.

CI boots a real 5-service DataHub on every push, seeds it, scans the live catalog, reconciles it against the offline pipeline, writes back, verifies from a separate client — then breaks the catalog on purpose and asserts the verdicts move as the model demands.

Two upstream PRs, one a bug fix in DataHub core. datahub#18996 (community-contribution): enabling profiling on a SQLite source crashed ingestion outright — a working recipe produced zero events the moment you asked it to profile. And datahub-skills#92, a datahub-economics skill; the Skills Registry has five skills and none is about cost.

What we learned

Test count is a vanity metric. So we corrupted our own code 364 ways — flipping comparisons, swapping + for -, doubling thresholds — and measured which corruptions our tests actually caught. 71.2% overall, 80.4% across the economic core.

It immediately found that cadence.py, the input to 92% of our headline, was the weakest module in the project at 46.7%. Nothing pinned the two-observation boundary. Nothing rejected a zero epoch. Five + → - corruptions survived in the cost apportionment, where subtraction yields negative costs the UI would render as savings. It's 93.3% now.

The most useful thing it found was in the tests themselves: three of the first nine tests we wrote to close those gaps passed under the mutant too. Writing a test that passes is easy. Writing one that fails when the code is wrong is the job, and only this tells you which you wrote.

And documentation drifts unless something checks it. Our recoverable figure dropped $185k after a bug fix and the writeup quoted the old number for days. tests/test_published_claims.py now parses every figure back out of the committed markdown and re-derives it. The build fails when the prose stops being true. It has caught us four times since, including twice in the last day.

What's next for Assay

  1. Read through DataHub's own MCP Server rather than only serving our own, so Assay composes with the agent surface instead of bypassing it.
  2. Warehouse billing adapters. ACCOUNT_USAGE and INFORMATION_SCHEMA.JOBS are ground truth for cost; the catalog is ground truth for blast radius. The right answer uses both.
  3. Land the cost-aspect RFC. DataHub ships a cost aspect on exactly one entity type, mlModel. Datasets and dashboards, where spend actually accrues, have nowhere to put it.
  4. Right-sizing as a pull request, not a recommendation — emit the dbt or Airflow schedule change directly.
  5. Calibrate usd_per_consumer_day from real incidents instead of asking the user for a number nobody knows.

Built With

python
typescript
react
webgl
datahub
graphql
model-context-protocol
docker
github-actions
sqlalchemy

Try it out

pip install -e .                     # no API key, no DataHub SDK needed
assay seed --datasets 1200 --out estate.json
assay scan --file estate.json        # the $2.6M estate above
assay ablate --file estate.json --budget 20
python scripts/mutate.py             # the mutation score above

Built With

Share this project:

Updates