dbt-testgen — 75 tests emitted, 139 refused

• 

• 

• 

Inspiration

I started where everyone starts: read the catalog's nullability flag, emit not_null. It produced 157 tests across 13 models and they all passed.

Then I counted the flag before trusting it.

Platform	NOT NULL	nullable
dbt	157	0
s3	102	0
snowflake	0	212
postgres	0	102

573 column records in DataHub's own sample, zero variance anywhere. customers.customer_id is NOT NULL according to dbt and S3 and nullable according to Snowflake and Postgres — the same physical column. The flag records which connector wrote it. Those 157 tests asserted nothing at all.

That is the failure mode of generated test suites, and it is not "too few tests". It is tests nobody can justify: they fail on correct data, someone deletes them, and the ones that mattered go with them.

What it does

dbt-testgen reads DataHub through the MCP Server and generates a dbt schema.yml where every test cites a sentence a human wrote — and reports every test it refused, with the evidence that was missing.

75 tests emitted. 139 tests refused.

The evidence comes from somewhere nothing else was looking. This deployment holds 18 DataHub Documents; ten describe columns in markdown tables:

| `promotion_id`   | FK → promotions. NULL means no promotion applied (~65% of orders). |
| `account_mgr_id` | FK → corpuser. Assigned sales rep (B2B accounts only). NULL for B2C. |
| `product_id`     | FK → products. Composite PK part 1. |

None of that is inferable from a column name or a type: primary keys, composite keys, foreign keys pointing at things that aren't tables, closed value sets, and which columns are legitimately empty.

The refusals

Count	Refusal
119	not_null — the flag behind it is a per-platform constant
13	not_null retracted — documentation says the column is nullable
3	relationships — nothing in the catalog declares a target
2	relationships — the documented target is not a table
2	unique — the documented key is composite

customers.account_mgr_id is documented FK → corpuser — a DataHub user, not a warehouse table. A name-matching generator points it at whatever the name resembles. This one reads the sentence and declines.

Proving the refusals rather than asserting them

Anyone can claim their refusals were right. dbt-testgen validate --with-refused adds the 13 retracted not_null tests back and runs the suite:

Done. PASS=101 WARN=0 ERROR=13 SKIP=0 TOTAL=114

The 101 evidenced tests still pass. Exactly the 13 retracted ones fail, and nothing else. The seed data contains real NULLs in precisely those columns because the documentation says they belong there.

The baseline run is PASS=101 ERROR=0. Injecting four defects — duplicate key, null key, invalid status, dangling foreign key — gives ten failures, because the documented relationships cascade downstream.

Where lineage does real work

order_details is a view over eleven models. DataHub records those edges, so a column arriving from order_items keeps what a human wrote about it there — which is how four of those 13 retractions land on a model with no documentation of its own.

Two rules keep it honest. A view inherits what a column means, never what role it plays in its source table's key: inventories.product_id being half a composite key says nothing about the same column in a flattened view. And when two upstreams document the column as referencing different things, nothing is inherited — the conflict is reported instead of silently resolved.

What it found that nobody asked for

Reading the whole catalog at once surfaces things a per-model linter structurally cannot. Six columns are described in a way matching no documented value:

order_status    description:  1=Pending, 2=Processing, 3=Shipped
                document:     Pending, Open, Shipped, Complete, Cancelled, On Hold
customer_class  customers:     Platinum, Gold, Silver
                order_details: Retail, Enterprise, Online
                document:      Premium, Standard

order_status is described as holding integers and documented as holding strings, three states versus six. A generator trusting the description emits accepted_values: [1, 2, 3] — wrong type and wrong cardinality. Every one of these descriptions is prefixed e.g., which is the tell: an illustration, not a contract. So none of them generates a test, the documented set is used instead, and the drift is reported.

Also: the schema's PII glossary terms tag eight more customers columns than the document that lists the table's personal data — including numeric surrogate keys. Over-tagging trains people to ignore the label.

How I built it

Python, reading DataHub through the MCP Server: search, list_schema_fields, get_lineage, search_documents, grep_documents, and add_structured_properties for the write-back, which publishes each model's evidenced coverage back to the catalog. Coverage ranges 16.7%–73.3% across the sample; the spread is the point, since coverage computed from a constant flag would be a constant 100%.

--publish-gaps closes the loop the other way: the refusal report is a work queue — every refusal is a test that would exist if somebody documented one column — so it is saved back into DataHub as a Document, linked to the models it concerns.

One call uses the Python SDK rather than MCP: creating the structured-property definition, because the MCP server can set a property's value but has no tool to declare the property. That is in the feedback submission.

Challenges

The enum parser is where most of the care went, and most of it is refusals: `1`–`5` is a range, `unit_price × quantity` is an expression, `orders.warehouse_id` is a reference, and anything prefixed e.g. is an illustration. A generated accepted_values missing one legal value is worse than no test at all. 79 unit tests, weighted toward exactly those cases.

The bigger challenge was accepting the first version was the generic one — 195 tests from schema flags and column names, all green, and I nearly shipped it. Measuring the flag instead of reading it is what turned a plausible generator into a defensible one, and it meant deleting most of what I had built.

Accomplishments I'm proud of

The entry makes one claim a judge can check in under a minute: run --with-refused and exactly the 13 retracted tests fail, nothing else. Everything else in this description is backed by a committed log in examples/.

What I learned

That the most valuable metadata in DataHub's own sample is the prose nobody reads, and the least valuable is the flag everybody reads.

What's next

Composite keys are reported but not enforced — dbt's built-in unique takes one column, so that needs dbt_utils.unique_combination_of_columns. And the refusal report, read the other way, is a ranked list of what is worth documenting first; turning that into a written backlog for the catalog is the obvious next step.

Built With

• datahub
• dbt
• duckdb
• mcp
• python