StewardAI

An AI agent that detects data governance gaps, reasons over DataHub context and lineage, and writes approved fixes back to the catalog.

Comment

Inspiration

StewardAI

Overview

StewardAI is an AI-powered data governance agent built with DataHub to automatically discover, explain, and resolve metadata and data-quality gaps in a data catalog.

Modern data catalogs often contain incomplete descriptions, missing ownership, undocumented sensitive data, and unreliable metadata. These gaps make it difficult for both humans and AI systems to understand and safely use organizational data.

StewardAI turns DataHub from a passive catalog into an active governance workflow.

The Problem

A data catalog can contain hundreds or thousands of datasets, but keeping metadata accurate is difficult.

A table may have:

  • No meaningful description
  • Missing ownership
  • Undocumented PII
  • Incomplete governance tags
  • Data-quality problems
  • Downstream dependencies that are difficult to understand

Finding these problems is only the first step. The more important question is:

Can an agent understand the context, explain the problem, propose a solution, and safely update the catalog?

That is what StewardAI is designed to do.

How StewardAI Works

StewardAI follows a governance loop:

Scan → Detect → Understand Context → Reason → Human Review → Write Back

  1. Scan

StewardAI scans a real DataHub catalog and identifies metadata and governance gaps across the catalog.

  1. Detect

Specialized detectors identify issues such as governance gaps, PII, ownership gaps, data-quality problems, and downstream impact.

  1. Understand Context

The system uses metadata and lineage relationships to understand why a finding matters instead of relying only on table names or isolated columns.

  1. Reason

An AI agent analyzes the detected issue and produces a proposed governance action with confidence and reasoning.

  1. Human Review

The proposed change is shown to a human before anything is written back.

  1. Write Back

After approval, StewardAI writes the approved metadata changes directly back to DataHub.

This creates a continuous feedback loop where the catalog becomes more useful after every approved action.

Example: customers_clean

One of the demonstrations focuses on customers_clean.

The table appears normal at first, but its upstream context reveals sensitive information that is not properly represented in its own metadata.

StewardAI follows the available context and lineage, identifies the PII-related governance gap, and produces a proposal explaining:

  • What was detected
  • Why it matters
  • What metadata should be changed
  • The confidence of the recommendation
  • The reasoning behind the recommendation

After human approval, the agent writes the approved metadata back to DataHub.

This demonstrates the difference between an agent that simply reports a problem and an agent that can actually help resolve it.

Impact Analysis

StewardAI also analyzes downstream dependencies.

For example, when analyzing orders_fact, the system traces downstream assets such as dashboards and shows what may be affected by a change.

This gives data teams visibility into potential impact before making governance or data changes.

Data Quality

StewardAI includes a realistic data-quality profiler that detects issues such as:

  • Missing values
  • Invalid values
  • Invalid email formats
  • Statistical outliers

For outlier detection, the profiler uses Median Absolute Deviation (MAD) rather than relying on mean and standard deviation.

This avoids the masking effect where extreme values can distort the statistics being used to detect those same outliers, particularly in small samples.

Evaluation

The project includes an evaluation harness and a generated catalog containing 24 tables across multiple business domains.

We validated the detectors against known ground truth rather than relying only on visual demonstrations.

During development, the evaluation process itself helped uncover issues in our tooling.

When expanding the catalog from 4 to 24 tables, the ground truth initially became stale for three tables that had already been correctly fixed through real agent write-backs. The evaluation harness exposed the discrepancy, leading us to update the ground-truth generation logic to distinguish between having any tag and having a PII tag specifically.

We also discovered a row-index bug in the realistic data generator. A corrupted row was always reported as row 0 instead of its actual index, initially making the quality profiler appear to miss eight issues. Fixing the generator revealed that the profiler was actually detecting 100% of the injected issues.

Finally, the first value-based phone detector was too broad and incorrectly classified an ISO-format signup_date column as a phone number. We fixed this by counting actual digits after removing punctuation and explicitly excluding ISO date formats.

These findings were discovered through cross-validation and testing, then fixed through targeted technical changes.

What We Learned

The project reinforced an important lesson about AI agents for data systems:

Context is what makes an agent useful.

An LLM can generate a plausible recommendation, but reliable data governance requires evidence from schemas, metadata, lineage, and actual values.

We also learned that evaluation infrastructure is part of the engineering process itself. Ground truth must evolve with the system, realistic test data must be validated independently, and detectors need adversarial cases rather than only clean examples.

Why DataHub

DataHub provides the foundation that allows StewardAI to reason about the data ecosystem rather than treating individual tables as isolated objects.

Metadata and lineage provide the context needed to identify relationships between datasets, understand governance issues, and make more informed recommendations.

Most importantly, StewardAI doesn't stop at reading the catalog. Approved agent actions are written back to DataHub so that the knowledge becomes part of the catalog itself.

Technology

StewardAI was built using:

  • DataHub
  • Python
  • AI/LLM reasoning
  • Metadata and lineage analysis
  • PII detection
  • Data-quality profiling
  • Automated evaluation
  • Ground-truth validation

Project Architecture

The core workflow is:

                    ┌───────────────┐
                    │    DataHub    │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │     Scan      │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │    Detect     │
                    └───────┬───────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │ Context + Lineage   │
                 └──────────┬──────────┘
                            │
                            ▼
                    ┌───────────────┐
                    │  AI Reasoning │
                    └───────┬───────┘
                            │
                            ▼
                    ┌───────────────┐
                    │ Human Review  │
                    └───────┬───────┘
                            │
                       Approved?
                            │
                            ▼
                    ┌───────────────┐
                    │ Write Back to │
                    │    DataHub    │
                    └───────────────┘
## What it does

## How we built it

## Challenges we ran into

## Accomplishments that we're proud of

## What we learned

## What's next for StewardAI

Built With

  • agents
  • artificial
  • data
  • data-quality
  • datahub
  • engineering
  • generative-ai
  • governance
  • intelligence
  • lineage
  • machine-learning
  • python
  • quality
Share this project:

Updates