DataSheriff

Autonomous Data Incident Commander powered by DataHub context graph.

Comment

Inspiration

Data teams at companies like Apple, Netflix, and Pinterest run complex data stacks where a single column rename or null spike can silently break downstream analytics dashboards and production ML models. Engineers waste $50,000–$500,000 per hour and 4–8 hours of manual firefighting jumping between dbt models, Airflow DAGs, Snowflake, and catalog tools just to locate the root cause.

We were inspired by a simple question: What if DataHub's metadata context graph wasn't just a static catalog for humans, but an active, self-healing command center for AI agents? What if an agent system could catch an anomaly, trace the full blast radius, write a production-ready dbt fix, open a GitHub PR, and write that operational knowledge back to DataHub so the system gets smarter after every incident?

What it does

DataSheriff is an autonomous, multi-agent Data Incident Commander powered by DataHub's context platform:

  1. 🐕 Watchdog Agent: Polls DataHub quality assertions, freshness signals, and SLAs every 30 seconds for anomalies.
  2. 🔍 Lineage Detective Agent: Traverses DataHub’s lineage graph via the MCP Server to map the full blast radius (datasets, dashboards, charts, ML model features) up to 3 hops downstream.
  3. 🔬 Schema Inspector Agent: Identifies exact breaking schema changes (e.g. column renames, type mutations, dropped fields) by comparing historical schema snapshots.
  4. 🤖 ML Impact Assessor Agent: Leverages DataHub's end-to-end ML lineage (training data $\rightarrow$ feature tables $\rightarrow$ models $\rightarrow$ deployments) to calculate risk scores for production ML models before prediction drift occurs.
  5. 🔧 Fix Synthesizer Agent: Generates production-grade dbt model fixes with backward-compatible aliases, Airflow defensive null-check DAG patches, and automatically opens a complete GitHub Pull Request.
  6. 💾 Knowledge Crystallizer Agent: Closes the Institutional Memory Loop by writing knowledge back into DataHub — creating Incident entities (urn:li:incident:...), tagging assets (datasheriff-incident-prone), adding "Known Failure Modes" to asset documentation, and setting custom properties.

How we built it

  • Agent Orchestration: Built with LangGraph using a state machine (IncidentState) that runs parallel investigation nodes (Lineage Detective + Schema Inspector).
  • DataHub Integration: Built with acryl-datahub Python SDK, GraphQL API, DataHub REST Emitter, and DataHub MCP Server.
  • LLMs & Reasoning: Powered by Claude 3.5 / 4.5 Sonnet with function/tool calling for strict, un-hallucinated schema logic.
  • - UI Dashboard: Built with Next.js 16, Vite, Tailwind CSS, Framer Motion, and WebSocket streaming for real-time visual agent activity traces, blast radius interactive graphs, and syntax-highlighted PR diff previews.
  • DataHub Skill: Authored a brand new open-source DataHub Skill (datahub-incident-triage) that enables any MCP-compatible agent (Claude Code, Cursor, Windsurf) to perform automated incident triage.

Challenges we ran into

  • Lineage Traversal at Scale: Graph line traversal can become noisy. We engineered depth-bound search algorithms (up to 3 hops) combined with entity categorization (filtering dataset vs chart vs ML model) to build crisp blast radius reports.
  • Ensuring Zero Code Hallucinations: Generating dbt models that don't break production required injecting actual DataHub GraphQL schema metadata directly into the Fix Synthesizer agent's tool context, ensuring strict type and column name matching.
  • DataHub Metadata Emitter Types: Programmatically constructing Pegasus/Avro schema classes (IncidentInfoClass, GlobalTagsClass, EditableDatasetPropertiesClass) with correct URN formats required deep diving into the DataHub SDK source.

Accomplishments that we're proud of

  • Closing the Institutional Memory Loop: Most agent projects only read from metadata catalogs. DataSheriff writes back to DataHub, turning every incident into permanent organizational knowledge.
  • End-to-End Automation: Watching DataSheriff detect a column rename and automatically open a fully-formatted, passing GitHub PR in under 3 minutes with zero human intervention.
  • Dual Offline / Online Architecture: Created a standalone Next.js UI dashboard with built-in mock data for instant testing, alongside a live dockerized backend.

What we learned

  • DataHub's context graph is the missing link for AI agents operating in enterprise data stacks. Without catalog context (lineage, assertions, ownership), text-to-SQL and code-generation agents hallucinate constantly; with pre-validated DataHub metadata, agent accuracy jumps from ~50% to >90%.

What's next for DataSheriff

  • Self-Applying Auto-Healing: Adding human-in-the-loop Slack approval buttons ("Approve & Deploy Fix to Staging").
  • Root Cause Causal Engine: Integrating dbt query log parsing to identify the exact commit or author that introduced a breaking migration.
  • Expanding DataHub Skills Registry: Submitting the datahub-incident-triage skill directly to the main DataHub core skills repository.

Built With

Share this project:

Updates