/DataSherlock-AI-Root-Cause-Analysis-for-Data-Pipelines

🔍 Inspiration

Every data team knows the 9 AM panic: a critical dashboard shows wrong numbers, and engineers begin the manual hunt — querying tables, reading Airflow logs, tracing lineage by hand. The average data incident takes 2-4 hours to diagnose, and most of that time is spent just figuring out where in the pipeline things broke.

DataHub already stores everything needed to solve this — lineage graphs, quality assertions, freshness SLAs, ownership. But it's passive: you have to know what to look for. We asked: what if an AI agent could do the detective work?

💡 What it does

DataSherlock is an autonomous AI agent that investigates data incidents using DataHub's metadata:

🔍 Search — Finds the affected asset in the DataHub catalog
🕸️ Trace — Maps the full upstream lineage (S3 → Snowflake → dbt → Dashboard)
✅ Interrogate — Checks quality assertions AND freshness at every node in the chain
🎯 Diagnose — Pinpoints the exact failure point with a confidence level
💥 Blast Radius — Maps every downstream asset affected
👤 Notify — Identifies asset owners with contact info
📋 Report — Generates a structured incident report with a concrete fix plan

Anyone can use it — no DataHub or SQL expertise required. Describe the problem in plain English, get a root cause in ~30 seconds.

🏗️ How we built it

Agent loop: OpenAI-compatible tool calling with Llama 3.3 70B on Groq — we chose Groq's LPU inference because our agent makes 5-7 sequential tool calls per investigation; sub-second token latency means a full investigation completes in under 30 seconds instead of minutes.
7 DataHub tools: search_datahub, get_upstream_lineage, check_data_quality, check_freshness, get_downstream_impact, get_asset_owner, get_schema_changes — built against DataHub's GraphQL GMS API, following the DataHub Skills pattern.
Sample datapacks: Demo runs against DataHub's healthcare and nyc-taxi sample datasets, which contain planted freshness and quality issues — DataSherlock reliably finds them.
Frontend: Streamlit app with live tool-call streaming, structured report rendering, and one-click incident samples.
Reliability: 13 automated tests covering every tool and the full mock investigation chain (13/13 passing).

📊 Demo Scenario (healthcare datapack)

Input: "Patient Admissions dashboard shows zero admissions today — we expect ~450."

DataSherlock's investigation: | Node | Platform | Status | |------|----------|--------| | S3 raw file | S3 | ✅ FRESH (uploaded 2:47 AM) | | ETL pipeline | Airflow | ❌ FAILED — Snowflake warehouse auto-suspended | | Snowflake raw table | Snowflake | ❌ STALE — 27 hours overdue | | dbt fact model | dbt | ❌ 3 critical assertion failures | | Looker dashboard | Looker | ❌ Showing wrong data |

Root cause identified: patient_admissions_etl failed at 3:14 AM with ConnectionError: warehouse COMPUTE_WH auto-suspended. Blast radius: 5 assets including an ICU capacity alert system and a readmission-prediction ML model. Fix + owners included automatically.

🚧 Challenges

Agent reliability: Early versions either under-investigated (checked only the dashboard) or looped endlessly. We solved this with a strict investigative methodology in the system prompt — deepest-node-first interrogation — which made root-cause identification deterministic.
Reasoning speed: A multi-step agent on standard LLM APIs took 2+ minutes. Moving inference to Groq cut end-to-end investigation time to under 30 seconds, which makes the tool feel like an investigator, not a batch job.

🏆 What we're proud of

The agent doesn't just say what broke — it produces a full evidence chain showing healthy vs. failed nodes, so humans can verify its reasoning rather than trust it blindly.
It correctly distinguishes the symptom (dashboard wrong) from the root cause (warehouse suspension 3 hops upstream) — the hardest part of incident response.

🔮 What's next

Proactive mode: continuously watch DataHub quality/freshness events and open investigations before users notice
Slack/PagerDuty integration for automatic owner notification
DataHub MCP Server packaging so Claude/Cursor users can invoke DataSherlock directly
Risk scoring for proposed schema changes ("if you rename this column, 47 assets break")