Enterprise AI Agent

An autonomous 8-agent engineering fleet using DataHub as an Intelligence Backbone to audit repos, detect schema drift, patch SQL, and write back enriched metadata via MCP.

Comment

💡 Inspiration

In modern enterprise data platforms, data drift and schema mismatches are silent system killers. A field renamed in PostgreSQL (analytics.users) silently breaks downstream Snowflake data warehouses (dw.dim_users) and BI dashboards—often going unnoticed until business stakeholders view broken reports.

While enterprise data catalogs like DataHub excel at indexing metadata, traditional workflows treat them as passive lookup directories. We asked ourselves: What if AI agents didn't just read DataHub, but actively leveraged DataHub as an Intelligence Backbone to detect failures, validate code changes, generate technical debt reports, and automatically update metadata back in DataHub?

This inspired us to build Enterprise AI Agent: an autonomous, 8-agent engineering platform that acts as an enterprise-grade data & software engineering team operating directly on your DataHub metadata graph.

🤖 What It Does & Architecture

Enterprise AI Agent orchestrates an autonomous fleet of specialized agents to bridge the gap between codebase git repositories and DataHub catalog governance:

[GitHub Event / User Request]
            ↓
     [Planner Agent]
            ↓
 [Repository Agent] ──> [Schema Agent (DataHub Audit)]
            ↓
 [AI Architect Agent ⭐ (Tech Debt Evaluation)]
            ↓
 [Validation Agent (SQL Fix)] ──> [Human Approval Gate]
            ↓
 [Documentation Agent (PR)] ──> [Knowledge Agent ⭐ (DataHub MCP Write-Back)]



### The 8-Agent Fleet:
1. **Planner Agent**: Decomposes natural language objectives into executable Directed Acyclic Graphs (DAGs).
2. **Research Agent**: Scans RFCs, team runbooks, and DataHub best practices.
3. **Repository Agent**: Parses repository ASTs and extracts SQL queries and data pipeline scripts.
4. **Schema Agent**: Audits dataset schemas cross-platform via DataHub GMS API.
5. **AI Architect Agent**: Computes repository Technical Debt Scores and specs microservices.
6. **Validation Agent**: Generates and tests schema-aligned SQL patches to fix broken pipelines.
7. **Documentation Agent**: Auto-generates GitHub PRs and interactive Mermaid ER & lineage diagrams.
8. **Knowledge Agent**: Performs bi-directional write-backs to DataHub GMS via official Model Context Protocol (MCP) server tools.

---

## 📐 Math & Technical Debt Formulation

To evaluate pipeline health and governance risks quantitatively, our AI Architect Agent applies mathematically rigorous scoring metrics supported by LaTeX:

### 1. Technical Debt Score \(S_{TD}\)

The overall repository Technical Debt Score is calculated dynamically as:

$$S_{TD} = \max\left(0, 100 - \left( w_1 \cdot D_{schema} + w_2 \cdot L_{missing} + w_3 \cdot O_{unassigned} \right)\right)$$

Where:
- \(D_{schema}\) is the total count of unaligned field types between source and target datasets.
- \(L_{missing}\) is the ratio of untracked lineage edges in the pipeline.
- \(O_{unassigned}\) represents datasets lacking assigned technical owners in DataHub.
- \(w_1, w_2, w_3\) are configurable governance weights (default: \(w_1 = 15\), \(w_2 = 10\), \(w_3 = 5\)).

### 2. Schema Mismatch Index \(I_{SM}\)

For inline verification of PostgreSQL to Snowflake pipeline mappings:

$$I_{SM} = \frac{N_{mismatched\_fields}}{N_{total\_fields}} \times 100\%$$

If \(I_{SM} > 0\%\), the **Validation Agent** automatically triggers code patch generation.

---

## 🛠️ How We Built It

- **DataHub Intelligence Backbone**: Integrated with DataHub GMS via the official `acryl-datahub` Python SDK (`DataHubGraph`) and evaluated using the `showcase-ecommerce` dataset pack (1,000+ entities).
- **Official DataHub MCP Server Write-Backs**:
  - `update_description`: Pushes AI-generated technical dataset documentation back to GMS.
  - `add_owners`: Automatically assigns technical maintainers (`alice@acme.com`).
  - `add_tags`: Tags audited datasets (`#AI-Audited`, `#Schema-Aligned`, `#Verified-Pipeline`).
  - `write_lineage`: Registers newly discovered dataset dependency lineages.
  - `save_document`: Persists architectural decisions & runbooks in DataHub.
- **Backend**: FastAPI, Python 3.11, LangChain / LangGraph for agent orchestration, and SQLGlot for SQL parsing.
- **Frontend**: Next.js / React dashboard displaying real-time agent log streams, ER diagram renders, and DataHub write-back activity logs.

---

## ⚠️ Challenges We Faced

1. **High-Cardinality Graph Traversal**: Querying line-level lineage across thousands of datasets in DataHub required optimizing graph queries to maintain low-latency response times for the Schema Agent.
2. **Atomic MCP Write-Backs**: Ensuring that multi-step write-backs (e.g., creating lineage + adding tags + writing documentation) complete atomically without creating partial or orphaned metadata states.
3. **Agent Alignment & Safety**: Ensuring that the Validation Agent only outputs syntactically valid, schema-matched SQL patches required strict guardrails and human-in-the-loop approval gates before issuing GitHub PRs.

---

## 🎓 What We Learned

- **DataHub is the ultimate AI Agent Backbone**: Agents become exponentially more effective when backed by a unified metadata graph rather than isolated source code repositories.
- **Bi-directional Metadata Flow**: Moving from passive metadata consumption to active AI write-backs creates a self-healing metadata ecosystem.
- **MCP Standard Power**: Using standard MCP tools allowed seamlessly plugging DataHub metadata capabilities into LLM reasoning loops.

Built With

Share this project:

Updates