StillTrue

• 

  01.Every number in this submission, with the denominator under it

• 

  02.One DataHub page: the schema says Airport_fee, the documentation says airport_fee

• 

  03.The gate in one run: no confirmation refused, edited-after-confirmation refused, then written and read back

• 

  04.DataHub's own timeline recorded the write, so you can verify it without trusting this tool

• 

  05.A fault no dataset page can render: fixing it moves 168 of 1,296,000 pixels

• 

  06.41 consecutive months of the TLC's real schemas, every month scored right

• 

  07.41 consecutive months of the TLC's real schemas, every month scored right

• 

  08.A coverage check finds the undocumented column and misses the rename

• 

  09.Four patches upstream, all open

The one-letter bug that nothing catches

In February 2023 the NYC Taxi & Limousine Commission renamed a column in its public trip records from airport_fee to Airport_fee. One letter, with no announcement and no deprecation notice. Every dbt model and every line of documentation that spelled it the old way kept running and quietly stopped meaning what it said. In January 2025 they added cbd_congestion_fee. The published data dictionary still doesn't mention it.

Nothing threw an error, and that is the problem. A description that has stopped being true fails silently, and the agents reading it have no way to know.

Who this costs

The team that pays for this is the one running the catalog. At DataHub's own April Town Hall, Pinterest described governing 400,000 tables down to 100,000 governed assets; in May, Miro reported that putting DataHub context between the query layer and the warehouse moved text-to-SQL accuracy from roughly 50% to roughly 90%. Both numbers rest on the assumption that what the catalog says is still true. DataHub's own audit framing is "how complete is our metadata," and its seven assertion types all describe the data. Nothing measures whether the prose is still accurate.

What StillTrue is

An agent whose subject is the catalog itself. It reads what humans wrote, compares it against what the schema and lineage actually do, and reports where the two have come apart, with citations. It never writes anything back without a human confirming that exact text.

On a full catalog scan of 77 datasets it reported 11 drifted descriptions, confirmed 12 as still true, and declined to judge 58 (81 checks in total), with zero false drift verdicts.

Every other number, with the denominator under it:

Why the abstentions are the point

The detector never guesses whether a word looks like a column. For a broken reference it asks DataHub's own change log what left this dataset: no entry, no claim. Requiring that evidence is what buys the zero false verdicts, and the same 25 tables produce 6 false ones without it. 58 of the 81 checks end in "not enough evidence," and that is the feature.

How it uses DataHub

• Reads through the Agent Context Kit: get_entities, list_schema_fields, get_lineage, get_dataset_queries, grep_documents.
• Uses the OpenAPI timeline as its evidence source. A schema break is only asserted when DataHub's own change log records the column leaving.
• Writes back into the graph through update_description, then re-reads the value out of DataHub before believing it landed. The correction shows up in DataHub's native UI and in DataHub's own documentation timeline as a DOCUMENTATION / MODIFY entry, so you can verify the write without trusting anything this tool wrote.
• Four patches went upstream during the hackathon, all of them still open. One is the Agent Context Kit fetching field descriptions and deleting them before the merge its docstring promises (datahub#18628), found by needing it to work.

The gate

A write with no confirmation is refused. Confirming one text and submitting another is refused too: the token is a hash of the exact content, so editing it afterwards voids the approval. One fix produces ten records in a hash chain, including both refusals, because a ledger of successful writes only would be a changelog. stilltrue verify recomputes the chain.

What is deliberately not claimed

• The official DataHub datapacks are the development set, so the 77-dataset scan measures scale. Generalisation is what the two holdout repositories are for.
• The 41-month TLC replay was once described as a holdout. The code changed after seeing its scores, so it is a third-party benchmark, and the timeline of who knew what when is written down in the repo.
• Both holdout repositories are dbt packages from one organisation, mined by the same script. That is what the 14 of 14 rests on.
• A third walk over 80 repositories in six other organisations found none that met the same frozen bar, and the null result is published rather than fixed by widening the rule. One of them produced 19 labelled positives against a threshold of 30; it was not scored and the threshold was not moved.
• Two detectors (freshness, semantic conflict) are unimplemented. The reason is written up as a measurement in docs/D2-FEASIBILITY.md.
• All four upstream PRs are still open and none has been reviewed.

Try it

EVIDENCE.md walks the whole chain in three minutes with nothing installed. If you would rather run it, one click opens a Codespace and make demo-from-cold takes it from nothing to the full loop (measured: 12 minutes 21 seconds on a 2-core container, most of it DataHub's images pulling).

Built With

• agent-context-kit
• datahub
• dbt
• docker
• openapi
• parquet
• python
• sqlite