LineageMedic

Renovate for breaking data changes.

Comment

Inspiration

A column rename is rarely hard in the producer repository. The real cost appears downstream: a dbt model owned by fulfillment, a finance mart with a public output contract, an Airflow SLA, and dashboards nobody remembered to check. Existing impact tools surface the blast radius but leave humans to coordinate and implement every repair.

We wanted an agent that does the next responsible thing: preserve compatibility, generate reviewable code changes for each owner, prove coverage, and leave the decision beside the affected assets in DataHub.

Why it is different

DataHub already answers what will break. LineageMedic answers what each owning team should merge, in what order, and why it is safe.

It does not perform a global string replacement. DataHub identifies the affected assets and their operational context; explicit repository contract bindings then produce three different repair policies for the same rename:

  • the producer keeps a temporary compatibility alias;
  • an internal fulfillment model migrates directly;
  • a board-reporting finance model reads the new field while preserving its public output contract.

Every code-bound lineage asset must receive a repair, and every generated artifact must pass deterministic validation before the campaign becomes reviewable.

LineageMedic is a bounded, tool-using code-generation agent rather than a chat response. It observes through DataHub MCP, plans an owner-specific migration, acts only on isolated repository copies, proves every generated change, and writes the approved decision back as durable DataHub memory. The planner is deliberately deterministic so it cannot invent repositories, skip a lineage-bound asset, or bypass the mutation gate.

What it does

Given shipping_country → country_code, LineageMedic:

  • calls DataHub MCP for column lineage, entity context, and observed queries;
  • ranks six affected assets across dbt, Airflow, Looker, and Power BI;
  • selects a compatibility-window migration plan;
  • edits four files across three copied repositories;
  • validates SQL syntax, dbt refs, contract compatibility, and lineage coverage;
  • produces an auditable JSON manifest and Decision document;
  • saves that document to DataHub only after explicit approval.

How we built it

The review application uses React, TypeScript, Vite, Fastify, and the official Model Context Protocol TypeScript SDK. A live adapter supports DataHub Cloud over Streamable HTTP and self-hosted DataHub through the official mcp-server-datahub stdio package. A sanitized live-run capture powers the public replay, while a checked-in fixture keeps the repair engine reproducible without credentials.

The repair engine is deterministic. It works on isolated repository copies, applies an explicit policy per lineage-bound asset, and blocks review status unless all validators pass. Full before/after patches and raw live MCP responses are retained in the run manifest.

DataHub MCP end-to-end proof

The public replay is backed by a sanitized capture from a real self-hosted DataHub OSS run through the official MCP server. LineageMedic called get_lineage, get_entities, and get_dataset_queries; used those responses to generate four repository repairs; and passed four deterministic validation gates. After a separate, explicit approval, it called save_document and persisted the migration Decision in DataHub as urn:li:document:shared-2daff315-7440-4ffb-b1db-2fe18c765c30. The repository includes the sanitized read manifest, persisted-URN writeback proof, DataHub screenshot, and the real GitHub repair PR.

Challenges

The hardest design problem was not generating a string replacement. It was deciding when a rename may propagate and when a public downstream contract must remain stable. DataHub's transitive lineage and query evidence provide that distinction. A second challenge was keeping the demo honest: recorded and live modes are visibly distinct, dbt ref resolution is not misrepresented as a warehouse-backed compile, and no PR or DataHub write is claimed until it exists.

Accomplishments

  • End-to-end repair of four real files across three repositories.
  • One-click visual demo plus a headless npm run demo path.
  • Live and fixture DataHub MCP adapters with identical semantics.
  • Explicit writeback approval aligned with the save_document safety contract.
  • Machine-readable evidence manifest and automated CI.
  • A sanitized self-hosted DataHub MCP capture with live lineage, entity, and query responses.
  • Checked-in generated SQL/YAML outputs that judges can inspect without running the application.
  • A real generated-repair pull request with a successful CI check and reviewable four-file diff.
  • An upstream contribution implementing the missing datahub-audit workflow in the official DataHub Skills repository.
  • A real approval-gated DataHub writeback with a published document URN.

What we learned

Metadata-aware code generation becomes much more useful when lineage is treated as an execution plan rather than a diagram. Ownership tells us who must review, observed queries reveal compatibility obligations, and a catalog document can carry migration memory across agents and teams.

We also learned that a trustworthy agent needs two independent gates: validation proves that a proposed change is structurally safe, while explicit approval controls external mutations. A larger model cannot replace either gate.

Try it

What's next

  • GitHub App support for opening coordinated draft PRs and tracking merge order.
  • Native dbt compile and warehouse dry-run adapters when profiles are available.
  • Generalized type-change and removal policies.
  • Reconciliation that re-queries DataHub after deployment and closes the compatibility window only when legacy usage reaches zero.

Built With

Share this project:

Updates

posted an update —

Live DataHub MCP proof is public

LineageMedic now has a complete, judge-verifiable read → act → write trail:

  • live get_lineage, get_entities, and get_dataset_queries responses;
  • four validated SQL/YAML repairs across three repositories;
  • a real, CI-checked repair pull request;
  • an explicitly approved save_document result with a persisted DataHub Decision URN.

Try the no-login replay · Inspect the live MCP read evidence · Inspect the approved writeback proof

Log in or sign up for Devpost to join the conversation.