DataGuard Agent

An AI-powered DataHub agent that detects metadata, lineage, governance, and data-quality issues, explains their business impact, and generates actionable SQL fixes.

Comment

Inspiration

DataGuard Agent

Inspiration

Modern organizations do not suffer from a lack of data—they suffer from a lack of trust in that data. Important datasets frequently have missing owners, incomplete descriptions, broken lineage, inconsistent classifications, and hidden quality problems. These issues are often discovered only after they affect a report, an operational decision, or a customer.

DataHub provides a strong foundation for discovering and governing data assets, but reviewing every dataset manually still requires considerable time and technical knowledge. We built DataGuard Agent to transform DataHub metadata into clear, prioritized, and actionable recommendations.

Our goal was to create an intelligent agent that does more than report that something is wrong. It should explain why the issue matters, identify the affected data assets, and recommend a practical correction.

What it does

DataGuard Agent acts as an AI-powered data-governance and quality assistant. It can:

  • Discover datasets and inspect their metadata through DataHub.
  • Review schemas, descriptions, ownership, tags, domains, and lineage.
  • Detect incomplete or inconsistent metadata.
  • Identify missing owners and sensitive-data classifications.
  • Detect broken or suspicious lineage relationships.
  • Execute safe, read-only PostgreSQL validation queries.
  • Identify null values, duplicate records, invalid relationships, and inconsistent business data.
  • Prioritize findings by severity and business impact.
  • Explain each issue in clear, non-technical language.
  • Recommend corrective and preventive actions.
  • Generate validation and remediation SQL for review.
  • Operate in demo mode without requiring external credentials.

The demonstration uses a realistic customs-clearance data environment containing connected operational datasets. DataGuard Agent analyzes these datasets and shows how governance and quality problems can affect payment reconciliation, shipment processing, inspections, regulatory reporting, and management decisions.

How we built it

We designed DataGuard Agent as a complete web application with separate presentation, analysis, metadata, and database layers.

Frontend

The interactive dashboard was built with Next.js and TypeScript. It allows users to:

  • Search and select datasets.
  • Start an automated analysis.
  • Review health and severity indicators.
  • Filter detected issues.
  • Examine business-impact explanations.
  • Inspect recommendations and generated SQL.
  • Explore dataset ownership, tags, schemas, and lineage.

Backend

The backend was developed using ASP.NET Core 8. It coordinates the complete analysis workflow and exposes structured API endpoints to the dashboard.

The backend contains:

  • A DataHub-compatible metadata client.
  • A deterministic validation-rule engine.
  • A read-only PostgreSQL validation service.
  • Severity and finding-classification logic.
  • SQL recommendation generation.
  • Optional OpenAI-powered explanations.
  • Demo data and fallback services.

DataHub and MCP

DataGuard Agent is designed to communicate with DataHub through the Model Context Protocol (MCP). This enables the agent to search for datasets and retrieve metadata such as schemas, ownership, tags, descriptions, domains, and lineage.

The integration layer separates DataHub communication from the analysis engine. This allows the project to run immediately in demo mode while remaining ready for connection to a live DataHub environment.

PostgreSQL

We created a five-table PostgreSQL customs dataset containing realistic relationships and intentionally introduced quality problems. The application uses read-only queries to validate the underlying data without modifying production records.

Analysis workflow

The agent follows a structured pipeline:

  1. Discover the selected dataset.
  2. Retrieve its metadata and lineage.
  3. Evaluate deterministic governance and quality rules.
  4. Run safe validation queries where appropriate.
  5. Collect supporting evidence.
  6. Classify and prioritize each finding.
  7. Explain its operational and business impact.
  8. Recommend a corrective action.
  9. Generate SQL for validation or remediation review.
  10. Present the results through the dashboard.

AI reasoning is used to enrich the explanations and recommendations, while deterministic rules provide consistent and verifiable evidence. This combination makes the results both intelligent and dependable.

Challenges we faced

One of the biggest challenges was combining deterministic validation with AI reasoning. Relying entirely on AI could produce inconsistent findings, while using only fixed rules could make the application too rigid. We solved this by allowing the rule engine to establish the evidence and using AI to explain the impact and recommend actions.

Another challenge was generating useful SQL safely. DataGuard Agent must never make uncontrolled changes to operational data. Therefore, database access is read-only, and remediation SQL is presented for human review instead of being executed automatically.

DataHub and AI integrations also depend on external services and credentials. To ensure that judges can evaluate the project immediately, we implemented a realistic demo mode that demonstrates the complete workflow without requiring private infrastructure.

Finally, we had to present complex governance, lineage, and quality findings in a way that both technical and business users could understand. We focused on clear severity labels, concise explanations, visible evidence, and actionable recommendations.

Accomplishments that we are proud of

We are proud that DataGuard Agent is more than a static concept. It is a working, end-to-end MVP with:

  • An interactive production-ready dashboard.
  • An ASP.NET Core analysis API.
  • A realistic PostgreSQL demonstration environment.
  • A DataHub MCP integration layer.
  • Deterministic governance and data-quality rules.
  • Optional AI-powered reasoning.
  • Safe SQL generation.
  • Dataset-lineage visualization.
  • Docker configuration and Windows setup support.
  • A fully functional credential-free demonstration mode.

The project demonstrates how an AI agent can move data governance from passive documentation to proactive detection, explanation, and resolution.

What we learned

We learned that data quality and metadata quality must be evaluated together. A technically valid table can still be dangerous if it has no owner, description, classification, or understandable lineage.

We also learned that AI is most effective when it reasons over reliable evidence. Deterministic checks provide consistency, while AI converts technical findings into clear business context.

Most importantly, we learned that trust is essential for any automated governance system. Users need to understand what was detected, why it matters, what evidence supports it, and what will happen before a correction is applied.

What is next for DataGuard Agent

Our next steps are to:

  • Add human approval workflows for proposed corrections.
  • Update DataHub metadata after an approved recommendation.
  • Create GitHub pull requests for version-controlled changes.
  • Support scheduled and continuous dataset monitoring.
  • Add notifications for newly detected critical issues.
  • Expand validation to additional databases and warehouses.
  • Introduce configurable governance-policy packs.
  • Track data-health and trust scores over time.
  • Measure downstream impact using complete lineage.
  • Learn from accepted and rejected recommendations.

Our long-term vision is for DataGuard Agent to become a trusted data-governance teammate that continuously monitors an organization’s data ecosystem, detects risks early, and helps teams resolve them before they affect operations or decisions.

Share this project:

Updates