๐ Inspiration
Every data team knows the 9 AM panic: a critical dashboard shows wrong numbers, and engineers begin the manual hunt โ querying tables, reading Airflow logs, tracing lineage by hand. The average data incident takes 2-4 hours to diagnose, and most of that time is spent just figuring out where in the pipeline things broke.
DataHub already stores everything needed to solve this โ lineage graphs, quality assertions, freshness SLAs, ownership. But it's passive: you have to know what to look for. We asked: what if an AI agent could do the detective work?
๐ก What it does
DataSherlock is an autonomous AI agent that investigates data incidents using DataHub's metadata:
- ๐ Search โ Finds the affected asset in the DataHub catalog
- ๐ธ๏ธ Trace โ Maps the full upstream lineage (S3 โ Snowflake โ dbt โ Dashboard)
- โ Interrogate โ Checks quality assertions AND freshness at every node in the chain
- ๐ฏ Diagnose โ Pinpoints the exact failure point with a confidence level
- ๐ฅ Blast Radius โ Maps every downstream asset affected
- ๐ค Notify โ Identifies asset owners with contact info
- ๐ Report โ Generates a structured incident report with a concrete fix plan
Anyone can use it โ no DataHub or SQL expertise required. Describe the problem in plain English, get a root cause in ~30 seconds.
๐๏ธ How we built it
- Agent loop: OpenAI-compatible tool calling with Llama 3.3 70B on Groq โ we chose Groq's LPU inference because our agent makes 5-7 sequential tool calls per investigation; sub-second token latency means a full investigation completes in under 30 seconds instead of minutes.
- 7 DataHub tools:
search_datahub,get_upstream_lineage,check_data_quality,check_freshness,get_downstream_impact,get_asset_owner,get_schema_changesโ built against DataHub's GraphQL GMS API, following the DataHub Skills pattern. - Sample datapacks: Demo runs against DataHub's healthcare and nyc-taxi sample datasets, which contain planted freshness and quality issues โ DataSherlock reliably finds them.
- Frontend: Streamlit app with live tool-call streaming, structured report rendering, and one-click incident samples.
- Reliability: 13 automated tests covering every tool and the full mock investigation chain (13/13 passing).
๐ Demo Scenario (healthcare datapack)
Input: "Patient Admissions dashboard shows zero admissions today โ we expect ~450."
DataSherlock's investigation: | Node | Platform | Status | |------|----------|--------| | S3 raw file | S3 | โ FRESH (uploaded 2:47 AM) | | ETL pipeline | Airflow | โ FAILED โ Snowflake warehouse auto-suspended | | Snowflake raw table | Snowflake | โ STALE โ 27 hours overdue | | dbt fact model | dbt | โ 3 critical assertion failures | | Looker dashboard | Looker | โ Showing wrong data |
Root cause identified: patient_admissions_etl failed at 3:14 AM with ConnectionError: warehouse COMPUTE_WH auto-suspended. Blast radius: 5 assets including an ICU capacity alert system and a readmission-prediction ML model. Fix + owners included automatically.
๐ง Challenges
- Agent reliability: Early versions either under-investigated (checked only the dashboard) or looped endlessly. We solved this with a strict investigative methodology in the system prompt โ deepest-node-first interrogation โ which made root-cause identification deterministic.
- Reasoning speed: A multi-step agent on standard LLM APIs took 2+ minutes. Moving inference to Groq cut end-to-end investigation time to under 30 seconds, which makes the tool feel like an investigator, not a batch job.
๐ What we're proud of
- The agent doesn't just say what broke โ it produces a full evidence chain showing healthy vs. failed nodes, so humans can verify its reasoning rather than trust it blindly.
- It correctly distinguishes the symptom (dashboard wrong) from the root cause (warehouse suspension 3 hops upstream) โ the hardest part of incident response.
๐ฎ What's next
- Proactive mode: continuously watch DataHub quality/freshness events and open investigations before users notice
- Slack/PagerDuty integration for automatic owner notification
- DataHub MCP Server packaging so Claude/Cursor users can invoke DataSherlock directly
- Risk scoring for proposed schema changes ("if you rename this column, 47 assets break")
๐ ๏ธ Built With
Groq ยท Llama 3.3 70B ยท DataHub (GraphQL GMS API) ยท Python ยท Streamlit ยท OpenAI SDK ยท pytest