LINEAGEPATCH

• 

  Implemented Streamlit guided demo: impact graph, Repair Contract, generated diff, and validator table.

• 

  Judge-first interaction design showing DataHub lineage, evidence, dry-run controls, and downloadable bundle.

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for LINEAGEPATCH

Inspiration

A one-line source rename can silently break dbt models, executive dashboards, quality checks, machine-learning features, and model inputs. Repository search finds text, but it cannot prove field-level lineage, ownership, governance meaning, or which downstream consumer matters most. We built LINEAGEPATCH so every proposed repair carries inspectable evidence.

What it does

The reproducible demo starts when raw.shop_orders.customer_id becomes buyer_id with the same type and nullability. LINEAGEPATCH:

• reads DataHub-shaped schema, fine-grained lineage, ownership, governance, documentation, and quality context;
• separates one direct from five indirect consumers;
• calculates a deterministic 106 / CRITICAL impact score from named evidence factors;
• creates a Repair Contract before generating code;
• generates a reversible buyer_id as customer_id compatibility alias in an isolated dbt workspace;
• proves the original project fails, then passes path, YAML, SQL, secret, dbt parse, compile, and build checks after repair;
• exports an Evidence Bundle with paths, factors, hashes, validators, provenance, and two dry-run DataHub write actions.

The guided fixture never mutates the source repository or DataHub.

How we built it

LINEAGEPATCH is an explicit state machine:

INGEST_CHANGE → FETCH_CONTEXT → ANALYZE_IMPACT → SELECT_STRATEGY → BUILD_REPAIR_CONTRACT → GENERATE_PATCH → VALIDATE_PATCH → BUILD_EVIDENCE → WRITEBACK_PREVIEW → HUMAN_REVIEW

Typed Pydantic models normalize facts. Deterministic breadth-first traversal computes dependency paths with hop and node caps. A versioned policy engine records every score factor. The generator copies only allowlisted files into a run workspace and creates a focused SQL/YAML patch. Validators run with fixed working directories, allowlisted argv arrays, filtered environments, timeouts, and no shell.

How DataHub is used

DataHub is the decision context, not a decorative database. The SDK adapter supports schema, dataset and column lineage, owners, domains, tags, glossary terms, assertions, documentation, and two approval-gated additive write types with re-read verification.

For the official agent path, LINEAGEPATCH includes a bounded stdio client for DataHub's official MCP Server. It verifies and calls only get_entities, list_schema_fields, and get_lineage, then stores hash-only tool receipts so raw metadata and credentials are not persisted. The repository also contains a DataHub Skills-compatible workflow.

Safety and verification

Metadata text is untrusted data, never instructions. Paths are sandboxed; symlinks, destructive SQL, sensitive files, and shell execution are rejected. Durable writes require a connected server, an approval-ready validation report, explicit human approval, stable idempotency keys, and post-write re-read proof.

The public fixture currently reports seven of seven blocking checks passing. The expected pre-patch dbt failure remains a visible non-blocking warning because it proves the scenario is genuinely broken.

Accomplishments

• Complete broken-before/fixed-after dbt fixture
• Six evidence-linked downstream impact paths
• Deterministic Repair Contract and Evidence Bundle
• 27 offline tests passing, with the live-DataHub test separated honestly
• Public Apache-2.0 repository, evaluator page, sample outputs, and 61-second video
• No external LLM key required for the core demo

Challenges and lessons

The hardest part was separating facts, proposals, and authority. Context and validator output are facts; the Repair Contract and patch are proposals; a human retains authority over durable repository and metadata changes. We also learned that honest degraded states—fixture, disconnected, warning, skipped, and not available—are product features, not demo blemishes.

Honest boundary

The public fixture proves the deterministic end-to-end product. The DataHub SDK and official MCP protocol paths are implemented and mock-tested, but a live server round trip is not claimed because Docker was unavailable on the build host. Generated patches remain proposals until reviewed.

What's next

Add removal, type, nullability, ambiguous multi-change, and incomplete-graph fixtures; capture a disposable live DataHub read/write/re-read proof; and extend Repair Contracts to Airflow, incident workflows, and multi-repository pull requests.

Built With

• datahub