DataHub Deprecation Janitor

• 

  roi

• 

  owner-ping

• 

  before-dataset

• 

  campaign plan

• 

  after-dataset

• 

  cleanup-pr

Inspiration

Enterprise catalogs quietly accumulate dead datasets — zero usage, no downstream consumers, weak ownership — while storage bills keep rising. DataHub already makes it possible to manually deprecate with evidence, and ships human-approved enrichment primitives. What was missing for real operations is the campaign layer: batch discovery, multi-signal evidence, a single human gate, multi-aspect write-backs visible in the UI, and operator-facing drafts for the people and code that still need cleanup.

That gap is what DataHub Deprecation Janitor is built to close.

What it does

Janitor is an agent that runs an autonomous deprecation campaign against a DataHub catalog:

1. Sweep datasets using usage + lineage + ownership signals
2. Assemble typed evidence into per-table dossiers
3. Build a Campaign Plan with dispositions (deprecate / notify_only / skip)
4. Stop at a human approval gate — nothing mutates until selected entries are armed
5. Write back to DataHub (deprecation lifecycle, deprecation-candidate tag, evidence description, linked evidence Document)
6. Emit offline artifacts: owner-ping drafts, cleanup-PR drafts, and a modeled cost/ROI estimate

Honesty invariants (by design):

• Catalog mutations require human approval of the Campaign Plan
• Owner pings are drafts only (never auto-sent)
• Cleanup PRs are drafts only (never auto-opened or merged)
• Cost/ROI figures are estimates (modeled projections, not measured cloud-bill savings)

Campaign differentiator

This is not “report + tag” and not single-entity enrich. The product is the campaign: batch sweep → evidence trail → plan → gate → multi write-back → operator drafts.

How we built it

• Hexagonal architecture (src/janitor/): pure core orchestration + ports + adapters so the safety gate is testable without a live GMS
• DataHub OSS surfaces: GraphQL / SDK for catalog reads, updateDeprecation, tags, descriptions, and evidence Documents
• Installable Agent Skill under skills/datahub-deprecation-janitor/ (npx skills add Argolinhas/datahub-deprecation-janitor)
• Deterministic messy fixture with ground-truth dead/alive identities for offline rehearsal and unit tests
• Judge path offline in ≤10 minutes: install → pytest → python scripts/rehearse_fixture_campaign.py → read committed examples/
• Optional live path against local DataHub quickstart for UI-visible write-backs

Sample outputs (no run required)

Committed under examples/fixture-primary/:

Offline fixture shape (when deprecate entries are armed for dry-run): candidates=7 · deprecate=5 · notify_only=1 · skip=1 · writebacks=5.

Challenges we ran into

1. Usage on OSS quickstart — sample packs don’t ship realistic usage timeseries; we seed/emit usage so deadness signals are real, not zero-filled noise.
2. Safety vs autonomy — campaign speed without silent catalog damage required a hard invariant: no approval → zero write-backs, enforced in core and covered first by tests.
3. Novelty adjacency — DataHub already has manual “deprecate with evidence” and enrichment skills; the build had to stay visibly above that via the campaign + drafts + gate story.
4. Demo honesty — judges may not run code; the video, README, and examples/ had to carry the full arc without overclaiming auto-send or measured savings.
5. OSS contribution — filed an upstream CLI packaging issue for experimental datapack (#18497); demo seeding uses the Python SDK instead.

Accomplishments that we're proud of

• Full campaign pipeline end-to-end (offline + live write-backs)
• Approval gate as a load-bearing product surface, not a slide
• Four golden examples judges can read without installing anything
• Installable DataHub Agent Skill packaging
• ≤3-minute demo video showing the messy → clean arc

What we learned

• Metadata agents win when they write back to the graph with evidence, not when they only narrate findings
• Human-in-the-loop is a feature for governance workflows — especially deprecation
• For hackathon judging, video + README + examples are first-class product surfaces

What's next

• Stronger code-graph cleanup PR generation beyond the seeded demo repo patterns
• Optional notification adapters (still draft-first / human-send)
• Packaging refinements for broader Agent Skills distribution
• Deeper cost models when dataset size is available from the catalog

Try it (≤10 minutes, offline)

git clone https://github.com/Argolinhas/datahub-deprecation-janitor
cd datahub-deprecation-janitor
python -m pip install -e ".[dev]"   # CPython 3.11
pytest tests/ -q
python scripts/rehearse_fixture_campaign.py

Skill install:

npx skills add Argolinhas/datahub-deprecation-janitor

• Repo: https://github.com/Argolinhas/datahub-deprecation-janitor
• Demo video: https://youtu.be/pRFcFLnTkPM
• License: Apache-2.0 · new work in the hackathon window (2026-07-06 → 2026-08-10)

Built With

• acryl-datahub
• agent
• apache-2.0
• data-governance
• datahub
• deprecation
• github
• graphql
• hexagonal-architecture
• lineage
• metadata
• pytest
• python
• usage-stats
• youtube