AgentsMeristem

When data breaks, agents just restate the symptom. AgentsMeristem walks the DataHub lineage to the root cause, writes it back as an incident, and runs as a local 7B any agent can call over MCP.

Comment

Inspiration

Data on-call is misery. A dashboard goes stale or fills with nulls, and someone greps a hundred tables to find where it actually broke. Two things make it hard: a downstream symptom is usually propagation, not origin, and the worst incidents are invisible in metadata — every stage reads "ingested just now" while the data itself is days behind. Today's data agents mostly read the catalog and rephrase the symptom. We wanted an agent that finds the cause and does something about it.

What it does

AgentsMeristem gives any AI agent a lineage-native root-cause sense. Given a symptom on a downstream asset, it walks the DataHub lineage graph upstream, probes the actual warehouse data at each stage (max timestamps, defect predicates — catching what metadata misses), localizes the break to the boundary where an unhealthy stage meets a healthy upstream, and writes the finding back to DataHub as an incident so the next person or agent inherits it. Same engine, many scenarios: freshness, data quality (with selective blast-radius), schema-drift impact, and provenance.

How we built it

It ships two ways. A capability layer any agent plugs into — an MCP tool, a DataHub Skill, and a Python SDK — and a first-party reference console (FastAPI + a no-build frontend) you can run and watch. The engine reads lineage via the GMS /relationships graph, reads schema and data signals, and raises real DataHub incidents. For the GPU angle we didn't just call a bigger API: we built DataHub-Agent-Bench (a reproducible root-cause benchmark) and fine-tuned a local Qwen2.5-Coder-7B + LoRA that distills DataHub context into a small, private, zero-egress model, trained on a SLURM cluster and served with vLLM.

Challenges we ran into

Real ones, all reproducible and documented in the repo: DataHub's file source on Windows parses the C: drive letter as a URI scheme (KeyError: 'Did not find a registered class for c'), so all ingestion had to run inside a Linux container; a CLI/server version mismatch broke datapack loading with a cryptic KeyError; searchAcrossLineage slice- timed-out on single-node OSS (we switched to /relationships); raiseIncident is documented as Cloud-only but actually works on OSS v1.5; and a local MySQL collided with the quickstart on port 3306.

Accomplishments that we're proud of

Two scenarios run end-to-end on real DataHub data with real write-back; all four MCP tools are functional; we verified the "any agent" claim over the actual MCP protocol; and we gave the DataHub-native version of the workflow back to the community as a new skill, datahub-triage (datahub-project/datahub-skills#32). On the benchmark, the fine-tuned 7B lifts the noisy buckets a signal-only rule can't touch (64% → 100%).

What we learned

The bottleneck for data agents isn't the model — it's context. Without DataHub context even a strong API model fails the hard cases; with it, a local 7B matches it. We also learned to ship the caveat: our benchmark's ~100% is in-distribution (the synthetic glossary is explicit), so it proves context is necessary and learnable into a small local model, not hard reasoning — and we documented exactly how to harden it.

What's next for AgentsMeristem

Harden the benchmark (rule-only glossary, more real-DataHub cases), serve the 7B into the live console, add structured-property write-back, and upstream an incidents tool for the MCP server (issue #136) so any agent can close the loop.

Built With

  • datahub
  • docker
  • fastapi
  • graphql
  • huggingface
  • lora
  • model-context-protocol
  • peft
  • python
  • qwen2.5-coder
  • slurm
  • sqlite
  • trl
  • vllm
Share this project:

Updates