Data Change Autopilot

AI agent that reads DataHub before changing your data.

Comment

Inspiration

Every data engineer has lived this: someone renames a column in a production table, and three dashboards silently break. Ownership is unclear, the impact is buried in dbt files and BI tools, and the tribal knowledge lives in one person's head. LLM agents can generate migration code, but without real context they hallucinate schemas and produce SQL that breaks on first run.

DataHub already holds the truth — schemas, lineage, ownership, view SQL. We built Data Change Autopilot to make an LLM agent use that truth.

What it does

A data engineer types a change in plain English (e.g. "rename email to contact_email"). Data Change Autopilot then:

  1. Reads DataHub for the real schema, lineage (table + column level), asset types, and view SQL
  2. Computes a deterministic impact score and risk level — explainable, reproducible, not produced by the LLM
  3. Generates the full migration — SQL for the source table, updated SQL for every downstream view/model, and rollback scripts
  4. Validates the code deterministically with SQLGlot against the DataHub schema
  5. Waits for human approval — nothing writes without a click
  6. Opens a real GitHub Pull Request — one clean commit, with the impact summary in the description
  7. Writes the change back into DataHub — column renamed in schemaMetadata, lineage repointed to the new column URN, and an annotation added linking to the PR

The loop is closed: the next person, or the next agent, inherits the truth.

How we built it

Stack: Python 3.11 · FastAPI · LangChain · Google Gemini · DataHub Python SDK · DataHub GraphQL API · SQLGlot · Next.js 14 · TypeScript · Monaco Editor · Docker Compose · GitHub REST API

Architecture:

  • The orchestrator uses the DataHub Python SDK (DataHubGraph) for reads and writes on core aspects, and DataHub's GraphQL searchAcrossLineage for lineage traversal — the same API that powers DataHub's own UI
  • All impact scoring is pure Python — deterministic and reproducible
  • All SQL validation runs through SQLGlot against schemas retrieved from DataHub — the LLM never validates its own output
  • The write-back uses read-modify-write on schemaMetadata, upstreamLineage, and editableSchemaMetadata, so we never blindly overwrite fields we didn't intend to touch

Challenges we ran into

The DataHub write-back was the hardest — and most rewarding — part. Three facts about DataHub's aspect model dictated the entire design:

  • UPSERT replaces the entire aspect. Emitting a naive reconstructed schemaMetadata would silently erase column types, descriptions, and the primary key. We had to read the existing aspect, mutate one field, and write it back.
  • EditableSchemaFieldInfoClass has no name field, so renames cannot be done through the editable layer. Renames must go through schemaMetadata itself.
  • PATCH exists for upstreamLineage but not for editableSchemaMetadata. For consistency (and to avoid subtle object-equality bugs in remove/add patches), we used read-modify-write everywhere.

Renaming a field also changes its URN. DataHub cannot delete a schemaField entity, so the old URN remains orphaned. We disclose this honestly in the README rather than hide it.

Accomplishments we're proud of

  • The write-back works and is verifiable. After approval, the column is really renamed in DataHub, the lineage really repoints, and the description really carries a link to the merged PR — visible in the DataHub UI.
  • We have proof. 18 integration tests failed after the write-back precisely because the column they looked for no longer exists in the catalog. That failure is the best possible evidence that the write actually happened.
  • 239 tests pass on host and in container, covering the full end-to-end path.
  • The examples/ folder contains real generated outputs so judges can evaluate quality without running the project.

What we learned

  • Metadata is only useful if agents use it and write to it. Reading is table stakes; contributing back is what closes the loop.
  • Deterministic scoring + LLM reasoning + deterministic validation is a design pattern that scales beyond migrations.
  • read-modify-write is boring and slow — and it is the right answer when the API is UPSERT-based.

What's next

  • Wire the MCP Server into the orchestrator (currently scaffolded)
  • Support column deletes and type changes end-to-end
  • Contribute a new DataHub Skill for column-level change proposals
  • Broaden connector coverage beyond the showcase-ecommerce datapack

Built with

python, fastapi, langchain, google-gemini, datahub, graphql, sqlglot, nextjs, typescript, monaco-editor, docker, github-api

Built With

Share this project:

Updates