DataHub Shepherd

A context-aware AI agent that safely retires data assets, migrates downstream consumers, and preserves every decision in DataHub.

Comment

Inspiration

Data teams rarely remove obsolete assets safely. A column rename, retired table, or pipeline migration can quietly break dashboards, transformations, data products, and production ML features. Engineers often know what they want to change but lack reliable context about everything downstream.

We built DataHub Shepherd around a simple idea: an AI agent should not make data changes from repository context alone. It should first understand the organization’s real metadata graph—schemas, lineage, ownership, governance signals, and ML dependencies—then act carefully and preserve what it learned.

Shepherd turns asset retirement from a manual investigation into a guarded, resumable workflow.

What it does

A data engineer gives Shepherd a mission in natural language, such as:

Deprecate raw.customers.email in favor of raw.customers.primary_email and migrate all safe consumers.

Shepherd then:

  1. Searches DataHub for the source asset and approved replacement.
  2. Reads schemas, ownership, governance metadata, and downstream lineage.
  3. Identifies affected tables, transformations, dashboards, pipelines, features, and models.
  4. Uses an AI decision engine to select a migration strategy and classify risk.
  5. Separates safe automatic actions from changes requiring human approval.
  6. Generates migration artifacts such as SQL transformations, compatibility aliases, dbt tests, and migration documentation.
  7. Records its evidence, decisions, blockers, and resume instructions in a persistent mission history.
  8. Writes a handoff document back through DataHub MCP so another engineer or agent can continue without repeating the investigation.

Shepherd supports more than column retirement. It can plan dataset migrations, investigate broken dashboards, remediate sensitive data, repair pipelines, and resume previously paused missions.

How we built it

Shepherd is a full-stack application built around a five-stage agent loop:

  • Discover: Resolve assets and trustworthy replacements using DataHub.
  • Trace: Map the downstream impact radius.
  • Plan: Let the AI decision engine choose a strategy and assess risk.
  • Act: Generate reviewable migration code and tests.
  • Record: Persist the result and write context back to DataHub.

The application uses:

  • DataHub MCP Server for catalog search, entity context, lineage, and document write-back.
  • OpenAI Responses API for evidence-grounded planning and tool selection.
  • Cloudflare Workers for the server runtime.
  • Cloudflare D1 for persistent missions, decisions, events, and resumable state.
  • React and TypeScript for the interactive application.
  • Human-in-the-loop approval gates for destructive, sensitive, ambiguous, or ML-impacting changes.

The model returns a structured decision containing:

  • Intent and migration strategy
  • Risk level
  • Supporting evidence
  • Proposed tool actions
  • Reversibility information
  • Generated artifacts
  • Handoff instructions
  • Whether human approval is required

Credentials remain server-side, and the application includes a deterministic guarded-policy mode so judges can explore the complete workflow without external credentials.

Challenges we ran into

Turning metadata into action

Reading lineage is useful, but this challenge called for an agent that performs real work. We designed Shepherd to convert catalog evidence into migration artifacts, approval decisions, and persistent operational state rather than stopping at an impact report.

Preventing unsafe autonomy

Data migrations can be destructive. A fully autonomous agent could easily cause more harm than it prevents. We created explicit approval boundaries for schema deletion, sensitive data, ambiguous replacements, and production ML dependencies.

Preserving context across agents

Most agent workflows lose their reasoning when a session ends. We introduced structured migration checkpoints containing completed actions, blockers, owners, evidence, and resume instructions. This lets the next person or agent inherit the operational context.

Building a reliable demonstration

A hackathon demo must remain understandable and testable even when third-party services or private catalogs are unavailable. Shepherd therefore separates its agent workflow from its integrations and includes a deterministic sample catalog that demonstrates the same decision and approval loop.

Integrating evolving APIs

Both agent and MCP ecosystems are developing quickly. We kept the DataHub and model integrations behind small server-side adapters so endpoints, authentication, and model choices can evolve without rewriting the application.

Accomplishments that we're proud of

  • Built a complete discover-to-write-back agent workflow.
  • Made DataHub’s context graph central to every migration decision.
  • Created a clear visual representation of downstream impact.
  • Added AI-driven strategy selection instead of relying on a scripted sequence.
  • Implemented human approval gates for high-risk actions.
  • Persisted missions and event history in a relational database.
  • Created resumable handoffs for future humans and agents.
  • Generated concrete, judge-reviewable migration artifacts.
  • Kept secrets out of the client and source repository.
  • Shipped a hosted application with an Apache 2.0 license and reproducible setup instructions.

What we learned

The most important lesson was that agent quality depends as much on context and control as it does on model intelligence.

Lineage alone is insufficient. The agent also needs ownership, schema details, governance signals, approved replacements, and operational history. Conversely, context without action creates another assistant that explains problems but does not solve them.

We also learned that safe autonomy is not binary. Shepherd uses graduated autonomy:

  • Read-only investigation proceeds automatically.
  • Reversible code generation proceeds automatically.
  • Ambiguous or destructive changes pause for approval.
  • Every action leaves evidence and a recovery path.

Finally, writing results back is essential. A useful organizational agent should improve the context available to every future user and agent—not repeatedly rediscover the same facts.

What's next for DataHub Shepherd

Our next steps are:

  • Add column-level lineage visualization from live DataHub responses.
  • Open real GitHub pull requests containing generated migrations and tests.
  • Integrate dbt, Airflow, Dagster, and Prefect repositories.
  • Notify affected owners through Slack or Microsoft Teams.
  • Add scheduled checks for deprecated assets that still have active consumers.
  • Support multi-agent collaboration between investigator, migration, testing, and governance agents.
  • Execute migrations in isolated preview environments before approval.
  • Compare pre- and post-migration lineage to verify that every consumer was handled.
  • Add rollback generation and automated recovery testing.
  • Contribute reusable retirement and migration skills back to the DataHub open-source ecosystem.

Our long-term goal is for Shepherd to become the trusted retirement layer for modern data platforms: an agent that understands the context, does the work, knows when to stop, and always leaves the organization smarter than it found it.

Built With

  • 2.0
  • agentic
  • apache
  • automation
  • context
  • datahub
  • dbt
  • devops
  • engineering
  • generative
  • governance
  • gpt-5
  • human-in-the-loop
  • lineage
  • management
  • mcp
  • metadata
  • mlops
  • model
  • openai
  • protocol
  • quality
  • react
  • typescript
  • workers
Share this project:

Updates