Self-Healing Semantic Agent

An autonomous, MCP-powered DataHub agent that scans metadata graphs, flags governance risks, and auto-generates Git PRs with dbt, Airflow, and ML guardrails.

Comment

Inspiration

It all started with a deep dive into mathematical models—specifically, how concepts like Information Entropy, Bayesian inference, and Cosine Similarity can elegantly strip away noise to reveal signal in massive amounts of unstructured data.

We wondered: What if we could apply these core mathematical and probabilistic concepts to solve the chaotic reality of modern enterprise data governance?

In large-scale data platforms like DataHub, metadata rapidly degrades as lineage breaks, orphan datasets accumulate, and sensitive PII spills into ML training pipelines. We were inspired to build an autonomous agent that doesn't just passively monitor metadata graphs, but actively calculates exposure risks and mathematically pinpoints bottlenecks to self-heal the platform.

What it does

We built the Self-Healing Semantic Agent—an autonomous agent for DataHub that goes beyond merely highlighting issues on dashboards by actively fixing them directly at the source.

Here is step-by-step how it works:

  1. Scans Data Graphs via MCP: We implemented a custom DataHubMCPEngine module powered by the Model Context Protocol. It connects to DataHub to gather full entity context—schema definitions, glossary terms, freshness SLAs, and upstream/downstream lineage.

  2. Analyzes Topology & Quantifies Risk: We use NetworkX to construct a directed dependency graph G = (V, E) and compute the Betweenness Centrality metric C_B(v) for each dataset:

$$C_B(v) = \sum_{s \neq v \neq t \in V} \frac{\sigma_{st}(v)}{\sigma_{st}}$$

This allows the agent to instantly pinpoint single points of failure and bottleneck nodes in the data architecture.

  1. Detects PII Leaks & Orphan Lineage: The agent automatically scans schema attributes and Glossary terms for sensitive PII markers, while simultaneously flagging orphan datasets (broken upstream links).

  2. Executes Metadata Write-Back: Based on the calculated risk score, the agent constructs MetadataChangeProposalWrapper payloads and directly applies governance tags (such as urn:li:tag:HighRiskPII) back into DataHub.

  3. Generates Automated Git Pull Requests: We integrated Groq LLM (Llama-3.3-70B) so the agent can draft remediation code on its own—dbt SHA-256 data masking models (_masking.sql), ML feature validation guardrails (_guardrail.py), and Airflow lineage DAGs (_lineage_dag.py). It then pushes these files to a remote branch and opens a production-ready GitHub Pull Request with a complete audit report.

How we built it

We built the Self-Healing Semantic Agent around three core architectural pillars aligned with the hackathon challenges:

  1. Model Context Protocol (MCP) Integration (Track 1): We implemented a custom DataHubMCPEngine adapter that acts as a bidirectional bridge. It queries GraphQL/REST metadata endpoints to pull schema states and emits MetadataChangeProposalWrapper payloads back to DataHub, dynamically tagging high-risk nodes (urn:li:tag:HighRiskPII).

  2. Graph Analytics & Topological Health (Track 2): To evaluate lineage health, we construct a directed graph G = (V, E) using NetworkX, where vertices V represent datasets/pipelines and directed edges E denote data flows.

We combine topological metrics with PII detection and freshness delays into a unified Risk Score:

  • Risk Score = 0.4 * C_B(v) + 0.4 * PII_score + 0.2 * Orphan_flag + 0.1 * Freshness_delay
  1. Automated LLM Remediation Engine (Track 3 & Code Generation): When an at-risk entity is identified, the system passes schema context to Groq's Llama-3.3-70B. The agent then automatically generates targeted remediation artifacts:
  2. dbt SHA-256 data masking models (_masking.sql)
  3. Python ML feature validation guardrails (_guardrail.py)
  4. Airflow DAGs to heal broken orphan lineage (_lineage_dag.py)

Finally, it pushes these artifacts to a remote Git branch and opens a fully formed GitHub Pull Request with an executive audit report.

Challenges we ran into

  • Taming Code & Graph Complexity: Translating abstract mathematical ideas into bulletproof Python logic wasn't easy. Wrangling directed graph topologies, missing lineage nodes, and runtime edge cases required extensive debugging sessions.
  • Graceful Degradation & Resilience: Real-world infrastructure fails. We had to ensure that if the DataHub GMS or external LLM API endpoints timed out, the agent gracefully degraded to offline evaluation templates without crashing the pipeline.
  • Formatting & Environment Conflicts: Balancing local development, terminal environments, Docker container configurations.

Accomplishments that we're proud of

  • Bridging Mathematics with Production Engineering: We successfully translated theoretical mathematical concepts and graph theory into practical, production-ready Python logic that quantifies real-world enterprise data risks.
  • True Closed-Loop Automation: We are particularly proud that our agent doesn't just send a passive alert to Slack; it autonomously decides on a fix, writes a code patch, opens a Git PR, and updates the metadata status in DataHub.
  • Resilience & Fault Tolerance: We engineered the system to gracefully degrade into offline evaluation modes using local sample context if external APIs or DataHub services become unreachable.
  • Full MCP Native Integration: We successfully implemented the new Model Context Protocol standard, making the agent modular and ready to interface with modern AI ecosystems.

What we learned

  1. The Power of the Model Context Protocol: MCP radically simplifies how autonomous agents interact with complex enterprise metadata platforms like DataHub.
  2. Applied Math in Data Engineering: Math isn't just theoretical—applying graph centrality metrics directly translates to finding single points of failure in massive data architectures.
  3. Closing the Governance Loop: True Data Governance isn't just alerting humans on a dashboard; it's giving AI agents the agency to generate code and open Pull Requests to fix problems at the source.

What's next for Self-Healing Semantic Agent

  • Bayesian Networks for Predictive Failure Analysis: We plan to integrate dynamic Bayesian models to calculate the posterior probability P(Failure | Stale Data), predicting pipeline outages before they happen.
  • Entropy-Driven Data Drift Detection: We want to leverage Shannon Entropy H(X) to track schema and data distribution drift, automatically generating Great Expectations assertion suites.
  • Interactive Human-in-the-Loop Arbitration: Adding a Slack/Teams integration where data engineers can review and approve agent-generated Pull Requests with a single click.
  • Multi-Catalog Support: Expanding the MCP adapter interface to natively support platforms beyond DataHub, such as Apache Atlas, OpenMetadata, and Snowflake Horizon.

Built With

  • agent-context-kit
  • datahub
  • datahub-docs
  • datahub-mcp-server
  • datahub-skills
  • docker
  • mcp
  • python
  • quickstart-guide
Share this project:

Updates