DataHub Aegis

Generate trusted SQL. Detect schema drift. Repair broken queries automatically. DataHub Aegis keeps analytics reliable as enterprise data evolves.

Comment

Inspiration

Enterprise Text-to-SQL systems face a problem that is much larger than generating syntactically valid SQL. Business terminology is private, datasets have different levels of trust, important rules are often buried in metadata, and schemas continue changing after a query is created.

A query that works today can silently become tomorrow’s production incident when an upstream engineer renames a column, changes a type or replaces a table. Most Text-to-SQL copilots stop after returning an answer. They do not remain responsible for what they generated.

That inspired me to build DataHub Aegis, a self-healing data agent that treats generated SQL as a protected artifact rather than a disposable response.

Aegis uses DataHub’s Context Graph to generate trusted analytics, verifies the SQL against a live database, monitors its dependencies, detects schema drift, repairs affected queries and writes the verified repair knowledge back into DataHub.

Its operating loop is:

GENERATE → VERIFY → MONITOR → DETECT → TRACE → REPAIR → VERIFY AGAIN → WRITE BACK

What it does

DataHub Aegis begins with a natural-language analytics request. Instead of sending that request directly to an LLM, it uses DataHub’s Agent Context Kit to retrieve authoritative context, including:

  • Relevant datasets and their schemas
  • Column descriptions and documented business semantics
  • Governance tags and deprecation status
  • Dataset ownership and platform information
  • Query-popularity signals and historical usage
  • Downstream lineage and dependency information

Aegis uses this context to generate schema-qualified SQL under a strict structured-output contract. It then applies a deterministic read-only policy that blocks destructive requests and multiple statements before database execution.

Safe queries must pass PostgreSQL EXPLAIN and execute inside a bounded, read-only transaction with a row limit and statement timeout. The user receives not only SQL and results, but also a decision receipt showing the DataHub operations performed, datasets selected, governance facts considered and live validation evidence.

Users can save verified queries for drift monitoring. When DataHub reports that an upstream schema has changed, Aegis identifies the protected queries in the impact set, validates their existing SQL and initiates a bounded repair workflow if they are broken.

In the demonstration, PostgreSQL physically renames:

Production.Product.ListPrice → Production.Product.CatalogPrice

The schema is re-ingested into DataHub and delivered through Aegis’s Metadata Change Log webhook boundary. Aegis compares the previous and current schema, identifies the probable rename and traces the affected protected query.

The old SQL fails live validation because ListPrice no longer exists. LangGraph retrieves the current DataHub schema and generates a candidate repair using CatalogPrice. That repair is not accepted simply because an LLM proposed it: Aegis reruns the safety policy, PostgreSQL EXPLAIN and bounded read-only execution. Only after every check passes does it update the protected query.

Every repair produces an auditable trail containing:

  • The detected schema change
  • The affected query and blast radius
  • Original and repaired SQL
  • Validation stages and retry count
  • Execution evidence
  • Write-back status and receipt ID

Finally, Aegis writes the verified receipt back into DataHub as durable organizational knowledge and applies an aegis-verified tag. Other analysts, applications and agents can inherit that result instead of rediscovering the repair independently.

How I built it

I built Aegis as an event-driven Python application with the following core components:

  • DataHub Context Graph as the authoritative metadata and lineage plane
  • DataHub Agent Context Kit for search, entity retrieval, schema expansion, query context and metadata mutation
  • LangGraph for generation, validation routing and bounded repair
  • FastAPI for query, system, monitoring and webhook endpoints
  • PostgreSQL for AdventureWorks data, live SQL verification and the persistent protected-query registry
  • Pydantic for strict generation, safety, workflow and repair contracts
  • SQL parsing and deterministic policies for read-only enforcement
  • An asynchronous architecture using HTTPX, SQLAlchemy and async model calls

The main query workflow is:

  1. Parse the user’s analytics intent.
  2. Retrieve and rank candidate datasets from DataHub.
  3. Expand the authoritative schemas and governance context.
  4. Generate structured, schema-qualified SQL.
  5. Reject unsafe or mutating statements.
  6. Validate the SQL using PostgreSQL EXPLAIN.
  7. Execute it in a bounded read-only transaction.
  8. Save the verified artifact and its DataHub dependencies.
  9. Write a verification receipt back into DataHub.

The repair workflow uses the same trust boundary. A proposed repair must satisfy the same structured contract and database checks as the original query before it can replace the protected SQL.

The demo runs against a real PostgreSQL AdventureWorks database and a local DataHub deployment. The interface displays backend-generated evidence rather than simulating workflow stages.

Challenges I ran into

One major challenge was separating plausible AI output from verified system behavior. An LLM can confidently produce SQL that references nonexistent or outdated columns. I addressed this by making live PostgreSQL validation mandatory and allowing registry updates only after the repaired SQL passes both EXPLAIN and bounded execution.

Another challenge was reliably identifying schema renames. A metadata system directly observes that one field disappeared and another appeared; it does not inherently know that the change represents a rename. Aegis compares the previous and current schema, uses the observed field and type changes as repair context, and proves the inferred mapping against the live database.

Local integration also presented practical challenges. DataHub’s PostgreSQL ingestion connector and the asynchronous application require different SQLAlchemy versions, so I isolated ingestion in a separate environment. I also built repeatable preflight, smoke-test, drift and restoration scripts to keep the demonstration deterministic.

DataHub Quickstart is not configured as an outbound webhook producer in this local environment. To preserve a production-realistic boundary without overstating the demo, the schema-drift control performs a real PostgreSQL migration, runs real DataHub re-ingestion and delivers an MCL-shaped event through the same webhook endpoint that a production DataHub webhook or Kafka consumer would use.

Accomplishments that I’m proud of

I am especially proud that Aegis goes beyond a conventional Text-to-SQL prototype.

It demonstrates a complete closed loop:

  • DataHub-grounded SQL generation
  • Deterministic rejection of destructive requests
  • Live database validation and bounded execution
  • Persistent monitoring of verified queries
  • Event-driven schema-drift detection
  • Dependency and impact analysis
  • Autonomous but bounded SQL repair
  • Before-and-after audit evidence
  • Verified knowledge written back into DataHub

The schema-healing demonstration changes the physical database rather than modifying only a mock metadata object. The saved query genuinely becomes invalid, the original SQL genuinely fails validation, and the repaired SQL must genuinely execute before it is accepted.

The result is an agent that is autonomous where useful, deterministic where safety matters and auditable throughout its lifecycle.

What I learned

The most important lesson was that metadata becomes significantly more valuable when it participates in execution rather than serving only as documentation.

DataHub’s Context Graph can provide the missing bridge between LLM reasoning and enterprise reality: which datasets exist, which ones are trusted, what their fields mean, how assets are connected and what changed.

I also learned that reliable agentic systems require multiple layers of trust. Structured model output is helpful, but it is not sufficient. A production-oriented workflow needs deterministic policies, authoritative context, live execution checks, bounded retries, persistent state and evidence that explains what the agent changed.

Finally, autonomous repair should never mean unchecked modification. Aegis separates candidate generation from acceptance: the model can propose a repair, but deterministic policies and the live database decide whether it is safe and valid.

What’s next for DataHub Aegis

The next step is to evolve Aegis from a local demonstration into a production-ready metadata reliability service.

Planned improvements include:

  • Consuming DataHub MCL events directly through Kafka or configured webhooks
  • Durable queueing, idempotency and dead-letter handling
  • Support for additional drift types, including type changes, table replacements and multi-column migrations
  • Human approval policies for ambiguous or high-impact repairs
  • Native integrations with dbt, BI dashboards and orchestration platforms
  • Dedicated read-only database roles and managed secret storage
  • Centralized observability, alerting and repair-quality metrics
  • Repair confidence thresholds and automatic escalation when evidence is insufficient
  • Multi-database and multi-dialect support

The broader vision is to make DataHub an active reliability plane for enterprise agents: not only helping them understand data, but allowing them to detect change, act safely, prove their work and share verified knowledge with every system that comes next.

Built With

  • agentcontextkit
  • datahub
  • langraph
Share this project:

Updates