DataHub AI Incident Copilot

An autonomous, lineage-aware AI agent that walks DataHub's metadata graph to diagnose broken data pipelines, writes postmortem reports & tags back to DataHub.

Comment

Inspiration

Data pipelines break silently. A table goes stale, an upstream schema shifts, or an ingestion job stalls — and nobody notices until downstream executive dashboards look completely wrong three hops away.

When data engineers investigate these failures, they manually reconstruct lineage, hunt down blast radius, and attempt to resolve the issue. Crucially, nobody writes down what broke or why in a durable place. The next time the failure recurs, another engineer starts from zero.

DataHub already stores the rich metadata graph (lineage, schema, quality, ownership) that makes these issues diagnosable. We were inspired to build an autonomous agent that not only automates root-cause diagnosis but also closes the loop by writing what it learned back into DataHub and generating preventive guardrails so the team and future agents inherit the fix.

What it does

DataHub AI Incident Copilot is an autonomous, lineage-aware AI agent that:

  1. Traverses DataHub's Graph: Connects to DataHub via the Model Context Protocol (MCP) to search entities, inspect schemas, and walk upstream and downstream lineage.
  2. Diagnoses Root Cause & Blast Radius: Pinpoints the exact breaking node (e.g., a dbt materialized view missing its 24-hour freshness SLA) and maps affected downstream assets (Looker explores, dashboards, and charts).
  3. Writes Back to DataHub (Closes the Loop): Tags broken datasets (add_tags), appends incident diagnosis to dataset descriptions (update_description), and files a formal postmortem report as a persistent document in DataHub's knowledge base (save_document).
  4. Generates Preventive Guardrails: Automatically writes concrete dbt sources.yml source freshness tests (warn_after: 12h, error_after: 24h) derived from the real database schema to prevent future occurrences.
  5. Provides a Claude Code-Style Interactive Terminal TUI: Features a rich interactive terminal shell (uv run datahub-copilot) with prompt history, slash commands (/status, /help, /clear), animated tool execution spinners, and syntax-highlighted output.

How we built it

  • AI Orchestration & LLM: Powered by Google Gemini 2.5 Flash (google-genai SDK) utilizing a native tool-use agent reasoning loop.
  • Metadata Integration: DataHub MCP Server (mcp-server-datahub) running over stdio subprocess, providing 18 read and mutation tools (search, get_lineage, list_schema_fields, add_tags, save_document).
  • DataHub Infrastructure: Local quickstart instance (datahub docker quickstart) loaded with the showcase-ecommerce pipeline datapack.
  • Python Environment & Packaging: Managed exclusively with Astral uv for dependency resolution and execution.
  • Terminal User Interface (TUI): Built with prompt_toolkit and rich to deliver an interactive terminal experience similar to Claude Code CLI.

Challenges we ran into

  1. Windows Drive Letter Path Parsing in acryl-datahub: When loading datapacks locally on Windows, drive letter schemes (e.g., C:\...) triggered key errors in datahub's ingestion filesystem module. We debugged and patched fs_base.py to correctly map single-letter drive prefixes to the file schema.
  2. GraphQL Timeout Bounds during Large Graph Ingestions: Ingesting large metadata packs into local Docker GMS instances led to GraphQL timeouts (default 30s). We adjusted the rest_emitter timeout configuration to 120 seconds to guarantee smooth tool calls across large lineage trees.
  3. MCP Mutation Tool Configuration: Ensuring write-back tools (add_tags, update_description, save_document) executed securely required configuring TOOLS_IS_MUTATION_ENABLED=true environment flags across stdio MCP subprocesses.

Accomplishments that we're proud of

  • Full Read & Write-Back Loop: Moving beyond read-only catalog lookup by actively updating DataHub metadata and saving persistent postmortems directly to DataHub's knowledge base.
  • Automated Code Remediation: Producing production-ready dbt sources.yml freshness guardrails derived from actual catalog schemas.
  • Polished Terminal Experience: Creating an interactive Claude Code-style terminal REPL with prompt autocompletion and live tool execution spinners.
  • End-to-End Execution: Running 100% locally against a real DataHub GMS backend with zero mock data.

What we learned

  • Power of MCP for Metadata: Standardizing catalog queries behind MCP tools allows LLMs to navigate complex graph structures with high precision.
  • Importance of Closing the Loop: Autonomous agents provide maximum ROI when their diagnostic insights are committed back into the enterprise knowledge base.
  • Schema-Aware Code Generation: Combining graph traversal with target code generation creates actionable artifacts engineers can merge immediately.

What's next for DataHub AI Incident Copilot

  • Automated PR Creation: Integrating GitHub/GitLab webhooks to automatically open Pull Requests with generated dbt freshness tests.
  • Slack & PagerDuty Alerting Integrations: Broadcasting incident summaries and DataHub document links directly to incident response channels.
  • Continuous Monitoring Daemon: Running an asynchronous background daemon that continuously sweeps DataHub for SLA drift and auto-files incidents before users report broken dashboards.

Built With

Share this project:

Updates