Inspiration
Data teams often discover broken data only when a dashboard looks wrong — long after the pipeline failed. Catalog metadata can still look healthy even when row-level freshness has silently stalled.
We built Data Incident Commander because the NYC taxi sample pipeline in DataHub has exactly this kind of invisible break: raw_trips stays current, staging_trips stops updating, and mart_daily_summary goes stale downstream — but ingestion timestamps do not reveal it.
We wanted an agent that does what a strong on-call data engineer would do: pull catalog context from DataHub, trace lineage to the earliest failing edge, ship a narrow fix, map blast radius, and leave a durable incident record in the catalog for the next person (or the next agent).
What it does
Given a broken dataset URN in a local DataHub instance, Data Incident Commander runs a six-stage autonomous loop:
Host probe — read-only SQLite check of actual row dates (because metadata alone hides the planted issue) Triage — MCP search, get_entities, schema lookup for owners, tags, and context Diagnose — MCP get_lineage and get_lineage_paths_between to find the root cause edge Codegen + PR — generates a narrow dbt freshness test and opens a real GitHub pull request Impact — downstream lineage to identify affected assets Write-back — MCP save_document to attach a structured incident report to the dataset in DataHub One command runs the full loop:
python -m agent.orchestrator Ground truth it finds: failed_transition: raw_trips -> staging_trips (9-day freshness gap propagating to mart_daily_summary).
How we built it
DataHub OSS via Docker quickstart for catalog, lineage, and document storage Official DataHub MCP Server (uvx mcp-server-datahub@latest) for live tool discovery and calls — we did not guess schemas; we dumped them live into docs/mcp_tools_schema.json Python agent with stage-specific prompts (agent/prompts/) and an async MCP wrapper (agent/datahub_mcp_client.py) LLM tool-calling loop with Anthropic Claude preferred and Google Gemini fallback (agent/llm.py) GitHub API (PyGithub) to seed a minimal dbt demo repo and open real PRs (agent/github_tools.py) NYC taxi static-assets datapack ingested locally for the broken pipeline scenario Host freshness probe because the MCP server has no SQL execution tool and the planted staleness is invisible in metadata Architecture:
broken dataset URN → probe → triage (MCP) → diagnose (MCP) → codegen (GitHub PR) → impact (MCP) → write-back (save_document)
Challenges we ran into
The break was invisible in metadata — all tables shared ingestion timestamps; we had to combine lineage with a row-level SQLite probe. nyc-taxi was not a registered datapack — we manually ingested from DataHub static-assets and ran lineage/metadata scripts. save_document did not appear until mutation was enabled — required TOOLS_IS_MUTATION_ENABLED=true. Tool naming differed from early docs — e.g. get_lineage_path vs actual get_lineage_paths_between; we verified live instead of assuming. Windows + MCP cold start — Inspector timeouts; we validated tools programmatically instead. GitHub fine-grained token scopes — had to use Contents API and correct repo permissions for seeding and PR creation. Gemini API quirks — model availability, rate limits, and no native role='tool' support required adapter logic and retries.
Accomplishments that we're proud of
Full closed loop on real DataHub — not a mock; live MCP tools, live GMS, live write-back Correct root-cause diagnosis — pinned raw_trips → staging_trips, not just the downstream mart symptom Real GitHub PR with a narrow, mergeable fix — dbt freshness test + schema update, not a vague rewrite Incident knowledge persisted in DataHub — structured markdown via save_document with related assets and PR link Judge-friendly repo — Apache 2.0 license, README with <10 min reproduction, and examples/ from a completed run Provider-flexible LLM layer — works with Anthropic or Gemini based on available API keys
What we learned
Metadata and data health are not the same thing — agents need both catalog context and row-level evidence for freshness incidents. MCP tool discovery beats documentation — live schema inspection prevented wrong assumptions about tool names and save_document parameters. Write-back is what makes agents useful long-term — opening a PR helps today; attaching an incident document in DataHub helps the next query tomorrow. Narrow fixes win hackathons and production — a targeted dbt freshness test at the failing edge is more credible than speculative pipeline rewrites. Agent design needs explicit trust boundaries — we only proceed to codegen/write-back when diagnosis is confirmed, and we keep evidence (probe JSON, lineage paths, PR URL) in the incident report.
What's next for Data Incident Commander
Slack / PagerDuty trigger — start the loop from an alert instead of a manual URN Multi-asset incidents — batch triage when several downstream marts fail together Airflow / dbt Cloud integration — patch DAG schedules or model configs, not only dbt tests Richer impact scoring — rank downstream assets by owner, SLA, and dashboard usage from DataHub metadata Incident deduplication — detect existing open documents before writing a new one Optional DataHub Skills layer — wrap repeated MCP patterns into reusable skills for faster agent runs
Log in or sign up for Devpost to join the conversation.