-
-
writeback_billing_model
-
writeback_retail_join_model
-
writeback_freshness_model
Inspiration
The hackathon's own rubric says it plainly: meaningful use of DataHub means writing back to the graph, not just reading from it. Most "AI pipeline generator" demos stop at reading a schema and hallucinating the rest. We wanted an agent that treats DataHub as a source of truth it's required to consult — schema, lineage, glossary terms, PII tags, freshness SLAs, ownership, quality scores — before it writes a single line of SQL, and then feeds back into that same graph what it actually built.
What it does
Give PipelineGuard a request like "create a model joining orders and customers." It:
- Queries the DataHub MCP Server, via a read-only session, for real schema, lineage (both upstream and downstream), glossary terms, tags, and governance metadata on the tables involved.
- Generates a grounded dbt SQL model via Gemini, including glossary-derived guard clauses and a PII safety report showing exactly which PII columns were kept out of the output.
- Optionally generates an Airflow DAG with two policy gates evaluated against that same DataHub context: a freshness/schedule check, and a quality-severity gate that reads the target asset's DataHub tags directly.
- Writes back to DataHub through a separate, explicitly mutation-enabled session — tags
the new asset
agent-generatedvia MCP'sadd_tags, and writes lineage edges from the source tables via the DataHub SDK. - Prints a PR-ready summary explaining which DataHub context drove each decision.
How I built it
Context Fetcher (read-only MCP session) -> Code Generator (Gemini, grounded prompt) + DAG Generator (policy gates keyed off real DataHub tags) -> CLI orchestration -> Write-back (mutation-enabled MCP session for tags, DataHub SDK for lineage). Verified against a local DataHub Quickstart instance (v1.5.0.6) loaded with the fiction-retail datapack (clean-schema baseline) and the healthcare datapack, which has planted data-quality issues specifically to test whether the quality gate catches what a naive generator would miss.
The clearest proof point: run the same request against two different target tables in the
healthcare dataset. One is tagged critical in DataHub — the generated DAG hard-halts
before running dbt. The other is tagged pii/internal but not critical — same generated
SQL, but the DAG only warns and continues. The DataHub tags are what changed the generated
code's behavior, not us.
Challenges I ran into
Our freshness check originally only compared in one direction — it caught a DAG scheduled more often than its SLA (wasteful) but silently passed the opposite case: a DAG scheduled less often than the SLA, e.g. weekly against a Daily SLA. That's the more dangerous real-world mistake, since it quietly serves stale data instead of failing loudly. We caught it during our own example-collection, fixed it the same day, and re-verified all three directions live.
We also hit a write-side surprise we hadn't planned for: DataHub's add_tags mutation
doesn't auto-create the target entity. Our first live write-back against a freshly
generated model URN failed with BAD_REQUEST "Entity does not exist" — we'd assumed
DataHub would stub the entity into existence on first write, and it doesn't. The same is
true for the agent-generated Tag entity itself on a fresh instance. We fixed this with
two small, idempotent bootstrap calls run before every write-back, rather than guessing at
a workaround.
And on the write-back path more broadly: we couldn't find an MCP tool for writing lineage
in the official mcp-server-datahub repo, so lineage write-back goes through the raw DataHub
Python SDK instead of MCP. Tags go through MCP's add_tags; lineage doesn't have an MCP
equivalent yet, as far as we found.
What I learned
DataHub's read-side MCP tools are genuinely good to build against — search, schema, and bidirectional lineage all did what we needed with no surprises. The write side needed more care: entities aren't auto-vivified, and lineage has no MCP tool at all yet. Both of those gaps are exactly what motivated the open-source contribution idea below.
What's next
- Contribute an MCP write tool for lineage upstream to mcp-server-datahub, so agents like this one don't need to fall back to the raw SDK for write-back.
- Validate generated DAGs against a real Airflow install instead of ast.parse.
- Extend write-back to DataFlow/DataJob entities once that write API is verified.
Honesty note
The DAG checks in this project are generation-time policy gates against DataHub data already fetched — not live runtime/warehouse checks. We don't have warehouse credentials configured and didn't find a reliable live signal exposed via MCP/GraphQL in our environment. We'd rather say that plainly than let a demo imply more than it does.
Log in or sign up for Devpost to join the conversation.