Groundskeeper

Writes dbt models from your real DataHub catalog, then proves the SQL is correct before you see it.

Comment

Inspiration

Ask any model for a dbt model and you get SQL that looks right. That is the problem.

Here is a real generation from this project's benchmark, asked for revenue by region with only the table names:

SUM(order_items.price * order_items.quantity) AS total_revenue
...
WHERE orders.order_date >= CURRENT_DATE - INTERVAL '30 days'

Two defects. There is no column called price; it is unit_price. And order_date is stored as VARCHAR in this warehouse, so that comparison throws at runtime. Neither is visible to a reviewer skimming a PR. Both reach production.

The gap is not that models write bad SQL. It is that nobody can tell when the SQL is wrong, because the thing that would say so, the real schema, the real types, the real governance rules, lives in the catalog and never reaches the model or the reviewer.

What it does

Groundskeeper generates a dbt model from the catalog, then refuses to hand it over until it has proved the model is correct.

  1. Ground pulls real schemas, native types, and column-level governance tags from DataHub over MCP
  2. Generate lets the model write SQL with that context in front of it
  3. Verify runs four gates, each able to block:
    • sql_analysis parses the SQL and resolves every column to a dataset, scope by scope
    • field_existence checks each column against list_schema_fields
    • governance compares sensitivity inherited from source columns against the output's own tags
    • compilation executes the query against DuckDB tables built from the catalog's types
  4. Repair turns a failure into a specific instruction and the model tries again
  5. Write back registers a verified model in DataHub with upstream lineage derived from the parsed SQL and any sensitivity it inherited

When the catalog is missing metadata, the run escalates instead of guessing. Retrying cannot invent a schema DataHub does not have, so it stops and asks.

The record of a blocked run

The record of one real run, with the catalog withheld from the model. field_existence rejects a column that does not exist; compilation rejects a date comparison because order_date is a VARCHAR. Both defects are in SQL that reads perfectly, and neither reached a reviewer.

Each gate carries its own history. The squares beside it are that gate's verdict on every attempt, so which gate failed, when, and whether the repair worked reads off one object. Here field_existence passed, then broke, and stayed broken.

A shipped run, registered in DataHub

A run that passed every gate, with the lineage it wrote back to DataHub.

Results

Eight realistic dbt tasks against the showcase-ecommerce datapack on a live DataHub. Every result decided by executing the SQL, not by reading it:

Agent First-try correct Shipped
Table names only, no catalog 1/8 (12%) 1/8 (12%)
+ DataHub schemas 7/8 (88%) 7/8 (88%)
+ verification and repair 7/8 (88%) 8/8 (100%)

DataHub's context graph is what moves 12% to 88%. The gates close the rest, and they are the reason those numbers can be stated at all: in the ungrounded arm they blocked seven broken models from reaching a reviewer.

The grounded arm is the verified arm's own first attempt rather than a separate generation, so the comparison carries no sampling noise. Per-task results are in examples/benchmark.json and the harness is scripts/benchmark.py, so any of this can be re-run.

How I built it

Python 3.11. mcp-server-datahub for catalog access, sqlglot for scope-aware column resolution, DuckDB as a throwaway warehouse built from catalog types, and the DataHub SDK for write-back. The generator works against any OpenAI-compatible endpoint.

The console is a single HTML file served by a standard-library HTTP server over SSE, because a judge should not have to install a web framework to watch it work. It is three surfaces rather than one screen, each answering a different question:

  • / what is happening now. Four gates sit in a fixed rail and change state in place, so a failure and its repair happen in the same spot on screen instead of scrolling away, and a gate's evidence stays visible while the repair is in flight.
  • /runs/<id> what happened, as a permalink. Every run is written to disk as JSON, folded together from the same events the browser saw, so a record cannot disagree with what was on screen. It has a "copy for a pull request" that emits the verdict table and every finding.
  • /benchmark why any of this should be trusted, per task rather than only in aggregate.

The benchmark, per task

Reading the per-task view rather than the summary is what makes the claim checkable: one task is blocked without the catalog, blocked with it, and shipped only after three verified attempts. That single row is the whole argument for the repair loop.

DataHub is the load-bearing dependency here, not a data source that could be swapped for a JSON file. The catalog's types build the warehouse the compilation gate executes against. The catalog's governance tags propagate to generated outputs, so PII cannot be laundered into an untagged model. Uncatalogued tables and unclassified columns escalate rather than pass, which turns metadata gaps into visible work instead of silent risk.

Challenges I ran into

Running it against a real catalog rather than fixtures exposed two bugs in my own design.

The governance gate escalated on columns that do not exist, which is the field-existence gate's job. Escalation halts the retry loop, so a repairable failure had become a dead end. And CTE names were being reported as missing catalog tables, which produced false escalations over SQL that was never wrong. Fixing the second one meant rewriting column resolution to work scope by scope via sqlglot's traverse_scope, so an unqualified column is attributed using its own select's tables rather than every table in the statement.

I also caught myself reporting a benchmark where the verified arm scored below the grounded arm, which is impossible by construction, since verified is grounded plus repair. It was sampling noise from running two independent generations. The arms now share one first attempt.

Making the console durable turned up three more, all of which only appear once something other than a browser uses the thing:

The SSE response sent Connection: keep-alive. Python's BaseHTTPRequestHandler reads that as an instruction to wait for another request on a connection that will never carry one, and an event stream has no Content-Length to say the body is over. A browser closes its own EventSource and never notices. curl, a script, or CI hangs until it times out. A run went from a 240-second timeout to a 13-second clean exit.

report.final is the last attempt, shipped or not, so a blocked run ends holding SQL that was rejected. The record view was about to label that "Final SQL", which reads as an endorsement of the exact query the gates had just refused. It says "Rejected SQL" now, explains that it was never handed on, and the pull-request copy never presents it as a result.

Run ids only carried second resolution, and history is ordered by sorting them. Two runs started in the same second fell back to sorting on a random suffix, which is no order at all. A test caught it; ids are now strictly increasing by construction instead of by hoping the clock is fine enough.

The same pass found a smaller honesty problem in the UI: an ungrounded run displayed "Grounded on" above the tables, contradicting the line directly above saying it ran without catalog schemas. The gates did use those schemas, the model just never saw them, so it now says "Checked against" and explains the difference.

Accomplishments that I'm proud of

The type defect. orders.order_date is a VARCHAR, and a model comparing it to a date produces SQL that reads perfectly and throws at runtime. No linter catches it, no schema check catches it, and few reviewers would. Executing the query against tables built from the catalog's own types catches it every time.

Then the same class of bug, in my own repository. After writing up the DataHub packaging bug I went looking for its shape here, and found it: the server resolved the console by walking up out of the package to the repository root, and the wheel only carries the package. Installed rather than cloned, Groundskeeper answered its own console with a 500 while the build and the install both reported success. CI now builds the wheel, installs it on its own, and reads the console through the installed package from outside the checkout, because building is not enough to catch a missing file that nothing imports.

And the numbers above check themselves. Six tests recompute the results table from examples/benchmark.json, so the headline cannot drift from the runs it came from. Two of them encode the invariant an earlier version of this benchmark broke: the verified arm cannot ship less than grounding alone, because it is grounding plus repair, and a result below it means the arms were sampled separately and compared across noise. The suite is 55 tests across Python 3.10, 3.11 and 3.12, and it runs with no DataHub installed, which CI enforces by failing if the SDK is importable.

What I learned

Grounding an agent in a catalog is worth far more than I expected: 12% to 88% on the same tasks with the same model. And verification is what makes that claim checkable rather than a promise.

Contributed back to DataHub

datahub-project/datahub#18633

package_data declared datahub.cli.resources, but the datapack agent-context file and the bundled offline registry live in datahub.cli.datapack.resources, a different package, so neither shipped in the wheel. datahub datapack --help raised FileNotFoundError whenever stdout was not a tty, which is exactly how an agent or CI job invokes it, and the offline registry fallback could never load. Found while building this, since datahub datapack load is the first command the hackathon's own resources page points people at.

What's next for Groundskeeper

Airflow and Dagster DAG generation using the same gates, and opening the generated model as a real PR with the evidence report as the PR body.

Try it

pip install acryl-datahub && datahub docker quickstart
datahub datapack load showcase-ecommerce

git clone https://github.com/calderbuild/groundskeeper && cd groundskeeper
uv venv --python 3.11 && uv pip install -e .

export GROUNDSKEEPER_API_KEY=...            # any OpenAI-compatible provider
export DATAHUB_GMS_URL=http://localhost:8080
export TOOLS_IS_MUTATION_ENABLED=true DATA_QUALITY_TOOLS_ENABLED=true

python scripts/console.py                   # http://127.0.0.1:8765
python scripts/benchmark.py                 # reproduce the table above

--ungrounded withholds the catalog schemas so the failure modes are visible directly.

Built With

  • datahub
  • dbt
  • deepseek
  • duckdb
  • mcp
  • python
  • sqlglot
  • sse
Share this project:

Updates