Datahub Health Agent

An AI agent that catches broken data pipelines, traces the root cause through lineage, and explains it in plain English.

Comment

Inspiration

Data teams often don't know when a dataset breaks, or why. I wanted to build something that doesn't just report a problem, but actually traces the cause and leaves the answer where the next person will find it.

What it does

An agent that monitors dataset health in DataHub — checking for missing documentation, missing ownership, and PII fields without documented access controls — traces the dataset's upstream lineage, and uses Claude to generate a plain-English health report explaining the findings. The agent writes that report back into DataHub as the dataset's documentation, so the next person or agent who opens it immediately inherits the finding.

How I built it

I used the DataHub Python SDK and Graph client to connect to a local DataHub instance, fetch a dataset's metadata, lineage, and schema, and check for governance gaps — including scanning column-level tags to detect PII fields. Findings get passed to Claude via the Anthropic API to generate a natural, non-technical explanation, which the agent then writes back into DataHub using the SDK's entity update methods. I tested it end-to-end across three datasets on three different platforms — S3, Postgres, and Snowflake.

Challenges I ran into

Testing across multiple datasets, not just one, surfaced two real bugs: my description check was reading the wrong metadata field, falsely flagging a documented dataset as missing docs, and my lineage summary had a hardcoded job name that reported the wrong upstream job for other datasets. I also hit a real-world constraint getting API billing set up, which pushed me to build the LLM step behind a clean interface so I could develop and test the rest of the pipeline before swapping in the live API call.

Accomplishments that I'm proud of

The write-back step and the PII detection. I didn't want to stop at printing a report to the terminal, so I made sure findings get inherited by the next person or agent, and added a real signal — unprotected PII fields — beyond just missing docs and ownership.

What I learned

How DataHub's metadata model is actually structured under the hood — the difference between raw ingested properties and editable/UI-entered ones, and how ownership and PII tags can live at different levels (dataset vs. schema field) depending on the platform.

What's next for Datahub Health Agent

A "scan all datasets" mode, checking for failing data quality assertions, and detecting inherited ownership from parent containers/domains, not just direct dataset-level ownership.

Built With

Share this project:

Updates

Submission history