DataHub Health Guardian

An agentic reliability control room using DataHub lineage, quality, and ownership context to stop healthcare data and ML pipeline issues before they impact clinical operations.

Comment

Inspiration

In clinical healthcare, bad data isn't just an engineering annoyance; it can directly impact patient outcomes (e.g., corrupting bed occupancy models, billing pipelines, or causing accidental PHI leaks). While traditional data catalogs are excellent passive books of record, they don't actively protect systems, and traditional monitoring tools produce fragmented alerts.

Our inspiration was to turn DataHub from a static catalog into an operational memory for the team. By putting multi-agent AI to work querying the live lineage graph, we can automatically triage pipeline issues based on downstream impact and write structured reports back to DataHub so the system continuously learns.

What it does

DataHub Health Guardian is a multi-agent reliability control room. A team of specialized agents monitors the health of healthcare data and ML pipelines:

  • Lineage Sentinel: Traces upstream schema changes downstream to identify exactly which dashboards, metrics, or ML models will be broken.
  • Data Quality Triage: Evaluates raw assertions (like negative billing amounts or invalid ages) and prioritizes them mathematically using a severity × lineage blast radius score.
  • Compliance Guardian: Intercepts pipeline changes, checks them against DataHub classification tags (PII/PHI), and alerts the team before sensitive data leaks downstream.
  • Orchestrator: Aggregates findings and translates them into a single, owner-aware P1/P2/P3 action plan.
  • Durable Write-back: The agents write their findings and recommended actions directly back to the affected DataHub entities' description fields via the Model Context Protocol (MCP) or REST, ensuring future engineers or agents start with the evidence instead of rediscovering it.

How we built it

  • Framework & Frontend: Built with Python and Streamlit for the interactive control room UI.
  • Context Platform & MCP: Grounded in the official DataHub instance. We launched and connected directly to @acryldata/mcp-server-datahub using standard stdio client sessions to invoke tools like get_entities, get_lineage, list_schema_fields, and update_description.
  • Testing Pipeline: Leveraged the official DataHub healthcare sample dataset (55k synthetic records on SQLite) with planted data-quality issues to test and run our agents locally.
  • AI Layer: Implemented a provider-agnostic LLM interface (supporting Gemini, OpenAI, Anthropic, and Groq) to synthesize natural-language action plans and formulate markdown reports.

Challenges we ran into

  • MCP Lifecycle under Streamlit: Streamlit normally executes synchronous page-level reruns, which caused event loop collisions when spawning long-lived async stdio-based MCP client connections. We solved this by using a dedicated synchronous background wrapper that handles the async event loops cleanly.
  • Resilience to SDK Versioning: The DataHub SDK has shifted its lineage enums and aspects across versions. We had to build defensive fallback mechanisms, normalizing raw text-to-JSON blocks returned by the MCP server to ensure it is backwards compatible.
  • Grounding Agent Reasoning: To prevent agents from hallucinating impact or priorities, we decoupled the calculations from the LLM. The agent logic mathematically traverses the lineage tree and computes blast radius scores, using the LLM strictly to write the final reports and synthesize the human-readable action plan.

Accomplishments that we're proud of

  • Closed-Loop Persistence: Achieving a true graph-native write-back where agent reasoning is durable. The write-backs are persisted directly inside DataHub as markdown, visible on the catalog's audit trial.
  • Zero-Setup Judge Demo: Packaging a fully functional, deterministic SQLite/Demo mode so judges can see the agents query lineage, trace PII leakage, and calculate quality blast radii instantly without needing a local DataHub instance up and running.
  • Standard-Compliant MCP: Fully wrapping the official DataHub MCP server package, making our solution portable to any agentic runtime that supports the Model Context Protocol.

What we learned

  • Metadata is the best grounding source for compliance: Agents do not need direct access to sensitive patient databases to triage pipeline failures. Grounding them on a metadata graph (schemas, tags, lineage, and ownership) preserves absolute HIPAA security while enabling deep operational reasoning.
  • Context-driven alert priority: Plain data quality assertions are noisy. Prioritizing alerts using the actual graph blast radius (how many models or dashboards are downstream) transforms how teams manage pipeline incidents.

What's next for DataHub Health Guardian

  • Real-time Webhook Triggers: Moving from a manual or polling mechanism to real-time webhook subscriptions that fire the Sentinel agents immediately when a schema change or failed assertion event occurs.
  • Confidence-Gated Auto-Remediation: Implementing auto-tagging or automatically proposing Git pull-requests to roll back breaking changes or lock down sensitive fields that have leaked downstream.
  • Multi-facility Orchestration: Aggregating pipeline health views across multiple federated hospital facilities using distributed DataHub catalogs.

Built With

Share this project:

Updates