Bloodhound

• 

  Bloodhound: an on-call reliability agent that tracks silent ML failures through DataHub lineage to the root cause.

• 

  The incident Bloodhound raised on staging_trips, the root cause: the table silently stopped 9 days behind its upstream.

• 

  Failing freshness assertion Bloodhound filed on the model's feature table; DataHub flags every downstream asset as unhealthy.

• 

  The trail it walks: model, training run, feature table, marts, staging, raw. First broken ancestor wins.

• 

  Same agent, different failure class: FIELD incident on raw_patients, 2.2% of rows violating the documented billing constraint.

Inspiration

Every ML team has lived this: the model keeps serving, the dashboards stay green, and days later someone discovers an upstream pipeline silently stalled, or contaminated rows have been flowing into the feature store the whole time. The failure was silent because catalog metadata said everything was fine: every table "ingested just now." Monitoring tools see statistics; none of them know the data stack's topology. DataHub does.

What it does

Bloodhound is an on-call agent that audits every ML model registered in DataHub:

1. SWEEP: walks each model's end-to-end lineage upstream (model → training run → feature table → marts → staging → raw) via DataHub's context graph
2. PROBE: compiles governance metadata into executable checks: a "Freshness SLA" glossary term becomes a MAX(event_time) probe compared across lineage edges; "must always be a positive number" in a column definition becomes a violation count that inherits across lineage to undocumented stages; all probes are templated, SELECT-only, and cite the metadata that justified them
3. DIAGNOSE: applies a first-broken-ancestor rule: one root cause with an evidence chain, not an alert per symptom, plus blast radius
4. RECALL: searches its own past RCA case files in DataHub, so recurring failures arrive pre-annotated with their history
5. ACT: writes everything back: a typed incident on the root-cause dataset, a failing assertion on the feature table (DataHub then flags "some upstreams are unhealthy" on affected assets), an RCA case document where every claim cites a URN and the probe SQL that produced it, blast-radius tags, and a generated fix bundle (backfill/guard/quality-gate SQL built from the evidence)

On the hackathon's own datasets it root-causes both planted failures with zero false positives: the nyc-taxi pipeline's silent 9-day staging stall (FRESHNESS, 90% confidence) and healthcare's contaminated rows (2.2% negative billing amounts, 1% NULL names, swapped dates) traced from raw intake into the model's feature table (FIELD, 90%).

How we built it

Python 3.11. The agent drives the official DataHub MCP Server through one tool surface with two drivers: LLM mode hands it to a PydanticAI agent (Anthropic/OpenAI/Gemini, pick via env var); --deterministic mode drives the same server programmatically, so judges can run the entire demo with zero API keys and identical results. Facts always come from the deterministic engine; the LLM may only phrase them (and its prose is fact-guarded against the evidence). Write-backs use the DataHub GraphQL/SDK shapes we verified live against OSS v1.5. The demo scenarios train real scikit-learn models registered through MLflow and ingested with DataHub's MLflow connector, so the ML lineage is genuine, not staged metadata.

Use of DataHub

Reads: search, multi-hop lineage, schema + glossary + tags (both baked and editable aspects), documents. Writes: incidents, external assertion runs, documents (agent memory), tags, all idempotent. Surfaces used: MCP Server, GraphQL API, Python SDK, MLflow connector, quickstart, the hackathon datasets, and DataHub Documents as a persistent, searchable agent memory, which is what makes the "next agent inherits the knowledge" loop real.

Contributed back

Building Bloodhound exposed two upstream gaps; we fixed both:

• acryldata/mcp-server-datahub#137: incident management tools (get/raise/update) for the official MCP server, with 15 tests and a live smoke proof; the proposal issue (#136) documents schema pitfalls we verified, including a docs bug (the incident enum is FIELD, not COLUMN)
• datahub-project/datahub-skills#31: the datahub-ml-rca skill: Bloodhound's methodology packaged so any coding agent can run the investigation with the DataHub CLI

Challenges

The planted issues are invisible in metadata by design: the datasets' own README says staleness "can only be detected by querying actual data timestamps." That forced the architecture's core idea: metadata tells you where to look and what to check, the data tells you the truth. False-positive discipline was the hardest part: freshness must be judged stage-to-stage (not wall-clock), volume anomalies must be interior holes (not ingestion-boundary tapers), and constraints must inherit across lineage to undocumented tables.

What we learned

The catalog is only half the truth. The planted failures were invisible in metadata, and the breadcrumbs the organizers left (SLA glossary terms, quality tags) only pay off if an agent treats them as executable instructions rather than annotations. We also learned to trust the live schema over the docs: two of our biggest time sinks (the FIELD vs COLUMN incident enum and the IncidentStatusInput shape) were settled by GraphQL introspection, and both findings went back upstream in our PR and proposal issue.

What's next

Watch mode in production (bloodhound watch), schema-drift detection as a third failure class, and moving the incident write-backs onto the MCP tools from our upstream PR once merged.

Built With

• datahub
• docker
• github-actions
• graphql
• mcp
• mlflow
• pydantic-ai
• python
• rich
• scikit-learn
• sqlalchemy
• sqlite
• typer