GhostBuster — Data-Platform Health Agent

AI cleanup agent that scans your DataHub context graph, flags orphaned, undocumented, and broken-lineage datasets, then writes LLM-generated fixes back to the graph.

Comment

Inspiration

Every data platform inherits the same mess: datasets with no owner, columns nobody documented, zombie tables kept alive "just in case", lineage pointing at upstreams that no longer exist. The chat agents everyone builds at an agent hackathon create new data. We wanted to build the opposite — an agent that cleans up the junk already living in your DataHub graph, and actually writes the fixes back instead of just reporting them.

What it does

GhostBuster scans your DataHub context graph and produces a concrete, LLM-written remediation plan:

  1. Scan — pull schemas, owners, lineage, glossary, tags, freshness and usage from DataHub.
  2. Diagnose — a fully deterministic rule engine flags 8 issue classes (missing owner, missing description, missing glossary, undocumented columns, orphan datasets, stale data, failing quality checks, broken lineage).
  3. Remediate — an LLM turns findings into a repair plan: add an owner, apply a glossary term, describe columns, deprecate, or fix lineage.
  4. Write back — the plan maps to DataHub mutations (corporateOwner, glossaryTerm, deprecation, field descriptions), closing the loop on the graph.

A healthy dataset passes with zero findings, so the agent never cries wolf.

How we built it

Python 3.10 standard library only, an OpenAI-compatible LLM layer (DeepSeek / any GPT-compatible proxy), and a thin datahub_client.py adapter with two backends: a bundled sample graph for zero-setup demos and a real DataHub backend over the GraphQL API. We seeded a live DataHub instance with five tables, injected a broken-lineage scenario (an upstream marked as removed), and watched GhostBuster detect it, write the owner/documentation fixes back, and re-scan clean.

Challenges we faced

Lineage edges to removed "ghost" entities are invisible in the default GraphQL query — we had to use includeGhostEntities: true to even see the break. DataHub's avro-based SDK rejected our seeding lineage; we switched to the GraphQL updateLineage mutation. Owner writes needed the right ownershipTypeUrn. Each of these became a fix that now lives in the repo's real-backend code path.

Built With

  • data-quality
  • datahub
  • datahub-graphql
  • deepseek
  • lineage
  • llm
  • metadata-governance
  • openai
  • python
Share this project:

Updates