Syntrace

• 

  Landing page hero

• 

  Dashboard: downstream impact table

• 

  Dashboard: pipeline all-green + drift cards

• 

  Dashboard: red/green diff open

• 

  PR body panel

• 

  Dashboard: write-back audit log

• 

  DataHub UI: stg_orders BEFORE

• 

  DataHub UI: stg_orders AFTER (money shot)

Inspiration

Every data engineer knows the 3 a.m. failure that starts with someone renaming a column. The upstream team ships a harmless-looking schema change, and hours later every dbt model and Airflow DAG that referenced the old name breaks at once. The metadata to catch this already exists in DataHub - versioned schemas and column-level lineage - but today a human has to read it, grep the pipeline repos, and hand-write the fixes. We wanted to close that loop: the same metadata that reveals the blast radius should drive the code that repairs it.

What it does

Syntrace is an autonomous agent that runs against a live DataHub Core instance - no mocked data anywhere:

1. Detects breaking schema changes by diffing DataHub's versioned schemaMetadata aspects (renames, drops, adds, type changes - renames matched by type + name similarity).
2. Traces the column-level blast radius through the official DataHub MCP Server (get_lineage with its column parameter), with a transparent GraphQL fallback.
3. Generates fixed dbt SQL and Airflow DAG code deterministically - renames propagated at any depth, dropped columns stubbed at the first hop to preserve downstream contracts.
4. Prepares a GitHub PR with a human-readable impact report (dry-run by default).
5. Writes the fix back to DataHub on four surfaces: remediated fine-grained column lineage, renamed downstream schema fields, a syntrace-remediated tag (attached via the MCP add_tags mutation tool), and a remediation note in each asset's documentation - so the catalog immediately reflects the healed pipeline.

Run it twice and the second run reports zero remaining impact - because the catalog was genuinely healed. It also ships a web dashboard, a CLI, a DataHub Skill (skills/syntrace-drift-remediation), and loads DataHub's official sample datasets.

How we built it

Python standard library only for the core (nothing to install), talking to DataHub over four surfaces: the MCP Server (stdio) for agent-context reads, GraphQL for entities and search, OpenAPI v3 for versioned-aspect reads and writes, and the DataHub Python SDK for seeding. The demo scenario is ingested as real metadata into a real catalog. 93 live-integration tests run against the running DataHub - zero mocks, enforced by a hygiene test.

Challenges we ran into

• Eventual consistency: DataHub's search index lags aspect writes; we had to add read-your-write barriers (async=false) and index-lag retries after reseeds.
• MCP mutations are opt-in: tools/list shows 6 read tools until you set TOOLS_IS_MUTATION_ENABLED=true (18 tools) - and there is no lineage-write tool, which is why write-back deliberately uses OpenAPI v3 alongside MCP add_tags.
• A self-mutating test environment: the agent's own write-back changes the catalog the tests read, so the suite restores pinned catalog state between runs.

What we learned

Metadata platforms aren't just documentation - with column-level lineage and versioned schemas, DataHub has everything needed to make pipelines self-healing. The gap was never the data; it was an agent willing to act on it and write its results back.

What's next

Real warehouse connectors beyond the demo scenario, sqlglot-based SQL rewriting for complex expressions, PR review-loop integration, and upstreaming our drift-remediation skill to the datahub-skills catalog.

Built With

• airflow
• css3
• datahub
• dbt
• docker
• github-api
• graphql
• html5
• javascript
• mcp
• nginx
• python
• rest
• ubuntu