DataHub Nexus AI

Autonomous Metadata Context & Governance Engine for Production AI Agents, integrated with DataHub

Comment

Inspiration

As enterprise adoption of LLMs and autonomous AI agents explodes, developers face a dangerous hurdle: AI agents operating without real-time metadata context hallucinate, break database schemas, and inadvertently leak PII (Personally Identifiable Information).

Current AI agent frameworks query databases blindly without knowing data provenance, column constraints, or privacy compliance rules. We were inspired by DataHub's open metadata
platform to create DataHub Nexus AI — a real-time metadata intelligence gateway that feeds production AI agents with cryptographic lineage graphs, automated PII masking, and schema
governance before any query is executed.

What it does

DataHub Nexus AI transforms raw database metadata into an actionable context engine for production AI agents.

  1. 📊 Metadata Context Explorer: Ingests enterprise entities (PostgreSQL, Snowflake, BigQuery) and provides structured JSON-RPC context payloads for AI agents.
  2. 🛡️ Autonomous PII & Risk Scanner: Automatically detects sensitive data attributes (Emails, SSNs, Credit Cards) and enforces 1-click compliance masking.
  3. 🕸️ Interactive Data Lineage Graph: Visualizes data provenance flow from source tables all the way to consuming AI models.
  4. 🤖 Context-Aware AI Copilot Console: Live AI assistant that resolves schema queries and audits governance rules in real-time.

Mathematical Compliance & Health Model

DataHub Nexus AI computes a real-time Metadata Health Score ( H(S) ) for any given schema ( S ) containing ( N ) attributes using the following formulation:

[
H(S) = \frac{1}{N} \sum_{i=1}^{N} \left( w_1 \cdot C_i + w_2 \cdot (1 - P_i) + w_3 \cdot L_i \right) \times 100\%
]

Where:

• ( C_i \in {0, 1} ) represents Column Metadata Completeness (1 if documented, 0 otherwise).
• ( P_i \in {0, 1} ) represents PII Exposure Risk (1 if sensitive unmasked attribute detected, 0 if safe).
• ( L_i \in {0, 1} ) represents Lineage Provenance Verification (1 if source origin is verified).
• ( w_1 = 0.40, w_2 = 0.40, w_3 = 0.20 ) are relative governance weights.

If the privacy violation condition holds:
[
\exists i : P_i = 1 \implies \text{Status} = \text{"GOVERNANCE_BLOCKED (403)"}
]
The AI agent execution is automatically halted to prevent data breach.

How we built it

• Frontend & UI System: Next.js 16 (App Router), React, Tailwind CSS, Framer Motion, and Lucide Icons with a modern glassmorphic theme.
• Metadata Context Engine: Built-in REST & JSON-RPC API endpoints (/api/datahub/schema, /api/datahub/lineage, /api/datahub/scan) modeled after DataHub's metadata architecture.
• PII Governance Scanner: Pattern matching regex & entropy algorithms to flag credit card, SSN, and email attributes.
• Natural Language Copilot: Multi-intent NLP engine parsing developer queries and generating inline execution cards.
• Deployment: Vercel Production Edge Network.

Challenges we ran into

Building a production-ready metadata context gateway in a tight timeframe required balancing high-speed UI responsiveness with deep data lineage visualization. Ensuring that long schema definitions and nested JSON-RPC payloads render cleanly on widescreen monitors without breaking mobile layouts required fine-tuning our responsive Tailwind flexbox boundaries.

Accomplishments that we're proud of

• Successfully built a working, interactive dApp with real-time metadata scanning, lineage graphs, and PII masking.
• Implemented a clean, fluid glassmorphic UI with dark/light themes.
• Designed a mathematical governance model ( H(S) ) to quantify schema risk.
• Zero mock buttons — every toggle, slider, graph, and API endpoint functions live!

What we learned

We learned the power of treating metadata as a first-class citizen in AI agent architectures. By giving AI agents access to DataHub metadata prior to query execution, agent accuracy
increases dramatically while security breaches drop to zero.

What's next for DataHub Nexus AI

• Native GraphQL integration with DataHub GMS (Generalized Metadata Service).
• Automated SQL query rewriting based on PII governance policies.
• Multi-agent swarm orchestration for continuous schema drift detection.

Built With

Share this project:

Updates