DataHub-Grounded Multi-Language Code Generation

Governance-aware Java and Rust generation grounded in DataHub MCP context.

Comment

DataHub-Grounded Multi-Language Code Generation

Turning governed DataHub metadata into verifiable Java and Rust applications through MCP-assisted coding agents.

Watch the 90-second demo · Source code and reproducible evidence

Inspiration

AI coding agents can generate convincing code, but they do not automatically understand an enterprise's real data contracts, governance rules, or security boundaries.

A prompt may describe what a developer wants, but it cannot reliably answer questions such as:

  • Which fields actually exist?
  • What are their native database types?
  • Which values are sensitive?
  • Does a documented relationship or lineage path really exist?
  • What should the agent do when required metadata is missing?

DataHub already contains this organizational context. We wanted to move it from passive documentation into the software-development loop.

Our goal was to demonstrate a practical workflow in which an external coding agent uses the DataHub MCP Server before generation, translates retrieved metadata into an executable TeaQL model, and produces Java and Rust applications whose behavior can be verified.

What it does

The project connects a coding agent to a self-hosted DataHub instance through MCP.

For the demonstrated payment dataset, the agent retrieves:

  • The real dataset URN and platform
  • Field names, native types, descriptions, and nullability
  • The dataset's sensitive-payment description
  • Available tags and ownership metadata
  • Upstream lineage

The agent then converts those facts into a TeaQL domain model.

The payment_account field is mapped from DataHub's VARCHAR schema into a TeaQL string property. Because the DataHub description identifies payment-account information as highly sensitive and requires masking, the agent adds:

_audit_mask_fields="payment_account"

DataHub returned zero upstream lineage and no documented foreign key or join connecting the payment and user datasets. The agent therefore keeps payment_account as a scalar field instead of inventing a plausible-looking user relationship.

The TeaQL generator runs through an internal loopback endpoint and produces Java and Rust domain libraries. Runtime tests then verify that safe audit outputs mask payment_account while retaining a non-sensitive field such as currency_code.

We also test the failure path. When DataHub returns Entity ... not found for an intentionally invalid URN, the coding agent refuses generation, returns empty business-field and relationship arrays, and asks for a valid dataset instead of fabricating a schema.

Architecture

Developer request
        |
        v
External coding agent
        |
        | DataHub MCP calls
        v
DataHub MCP Server ----> Self-hosted DataHub
        |                   schema, descriptions,
        |                   tags, ownership, lineage
        v
TeaQL model
        |
        | internal loopback endpoint
        v
TeaQL generator
        |
        +---- Java domain library + masking adapter/test
        |
        +---- Rust domain library + safe-event masking test

MCP is used by the coding agent at design and generation time. The generated Java and Rust applications do not require an MCP connection at runtime.

How we built it

We deployed DataHub OSS with Docker and ingested a payment dataset into the catalog. A small Python JSON-RPC client records reproducible MCP initialization, tool calls, arguments, timestamps, and sanitized responses.

The coding-agent workflow uses get_entities to obtain schema and governance context and get_lineage to determine whether a relationship is supported. Each resulting TeaQL decision is documented so reviewers can trace DataHub facts into the model and generated code.

The model is evaluated before generation. Java and Rust outputs are generated from empty directories, compiled, and checked for repeatability. Separate runtime tests send synthetic payment data through the safe audit paths and assert that the sensitive value cannot appear in the sanitized event or formatted output.

The repository contains the captured MCP JSONL, normalized DataHub context, generated model, decision log, representative generated sources, build logs, masking tests, negative-agent test, and a context-to-code evidence map.

Challenges we faced

Distinguishing facts from plausible assumptions

A payment account looks like it might reference a user, but DataHub returned no lineage, glossary term, foreign key, or documented join supporting that relationship. We removed the unsupported relation rather than allowing the agent to guess.

Turning governance text into runtime behavior

The sensitivity requirement was present in the dataset description while the captured field-tag arrays were empty. We had to preserve that provenance, map it to the correct field, and avoid claiming that a PII tag existed when it did not.

Separating safe analytics logs from controlled forensic logs

TeaQL has raw audit paths intended for controlled environments and sanitized paths intended for downstream analysis. We documented this boundary and verified only the safe Java and Rust paths used by the demonstration.

Producing reproducible evidence

Coding-agent interfaces do not always expose raw MCP traffic. We created a timestamped, sanitized JSONL evidence path so judges can inspect the actual requests and responses without relying only on screenshots or claims.

What we learned

The absence of metadata can be as important as its presence.

A lineage result of zero is not useless: it prevents an agent from generating an unsupported relationship. An entity-not-found response should stop generation rather than invite the model to complete the missing schema from prior knowledge.

We also learned that governance becomes much more valuable when it is translated into an executable constraint. In this project, DataHub context does not merely appear in documentation—it changes the TeaQL model, generated metadata, and observable Java and Rust audit behavior.

Hackathon scope

The TeaQL generator, Java and Rust runtimes, generated-code templates, and large ERP sample existed before the hackathon.

The hackathon-specific work is the DataHub MCP integration, payment scenario, context-to-model workflow, internal generation path, Java and Rust masking integrations and tests, negative missing-context behavior, and reproducibility evidence.

Antigravity performed the initial MCP-assisted implementation. OpenAI Codex Desktop subsequently reviewed and corrected the workflow, reran MCP interactions, regenerated outputs, implemented and verified masking behavior, and collected the final evidence.

What's next

We would like to create a compact DataHub “Agent Context Bundle” that returns schema, classifications, glossary terms, lineage, ownership, freshness, and explicit metadata-completeness states in one provenance-aware response.

We also plan to expand the policy mapping beyond audit masking to cover retention rules, authorization requirements, data residency, and other governance constraints—while continuing to require executable tests for every generated policy claim.

Built with

  • DataHub OSS / Core Platform
  • DataHub MCP Server
  • TeaQL
  • Java and Spring Boot
  • Rust and Cargo
  • Python
  • Docker
  • React, TypeScript, and Remotion
  • ElevenLabs
  • Antigravity
  • OpenAI Codex Desktop

Built With

Share this project:

Updates