DataHub Agent Crew

5 autonomous AI agents that find governance gaps in live DataHub instances and write verified fixes directly back through MCP, with zero human intervention and live schema validation.

Comment

Inspiration

Data governance in enterprise metadata platforms like DataHub often relies on manual reviews, leading to unowned datasets, stale descriptions, missing compliance tags, and broken lineage. We wanted to build a crew of autonomous AI agents that don't just alert humans, but actually fix governance gaps in real time with zero human-in-the-loop intervention.

What it does

DataHub Agent Crew is a five-agent pipeline that reads a DataHub metadata graph via MCP (mcp-server-datahub), identifies governance flaws, triages them, plans compliant updates, and writes fixes back to DataHub with post-write verification against the live entity.

  1. Investigator: Scans the metadata graph to locate unowned assets, missing compliance tags, stale descriptions, and lineage gaps.
  2. Analyst: Triages and prioritizes gaps based on severity.
  3. Strategist: Proposes exact metadata fixes and tag assignments.
  4. Regulatory: Audits proposed changes against compliance constraints before authorizing writes.
  5. Codeband: Executes updates to DataHub via MCP mutations and runs end-to-end write verification.

Inter-agent communication runs over a lightweight local SQLite bus using standard @mention routing—no external brokers or token rails required.

How we built it

  • Architecture: Built in Python using a long-running, multi-threaded agent pipeline polling a local SQLite message queue.
  • Metadata Integration: Uses mcp-server-datahub inside a unified MCP client wrapper (src/datahub_mcp.py) for read/write access.
  • Orchestration: Integrated src/mcp_hooks.py to route read-side graph searches to Investigator and write-side mutations to Codeband.
  • Write Verification: Implemented live entity re-fetching post-mutation (update_description → EditableDatasetProperties) to guard against silent API failures.

Challenges we ran into

  • Schema Validation & Silent Failures: Uncovered cases where API calls returned successful statuses without persisting changes, requiring us to add strict write-verification checks directly against live entity fetches.
  • Concurrency & Rate Limits: Resolved rate-limit exhaustion during initial graph scans by tuning polling frequencies and message bus throughput.
  • Edge-Case Parsing: Handled schemaField URN parsing bugs and reasoning-model content leaks to ensure clean inter-agent handoffs.

Accomplishments that we're proud of

  • Closed-Loop Automation: Successfully built a full loop where 5 specialized agents hand off tasks seamlessly to repair a live metadata graph.
  • Schema Safety: Confirmed via a real test case (an invalid DELIVERY_TYPE field write against our Geographic_Measures dataset) that DataHub's schema validation correctly rejects malformed mutations before they reach the graph — the system fails safe instead of writing garbage.
  • Zero Bloat Infrastructure: Kept the messaging stack entirely local and fast using SQLite and @mention routing.

What's next for DataHub Agent Crew

  • Expanded Entity Support: Extend mutations beyond datasets to pipelines, charts, and dashboards.
  • Real-time SSE Dashboard: Build a live streaming UI to visualize agent handoffs and write verifications visually in real time.

Built With

Share this project:

Updates