ContextForge Agent

A DataHub-aware agent that reads schemas, lineage, owners, and quality rules before generating mergeable dbt, Airflow, and catalog write-back artifacts.

Comment

Inspiration

Most AI coding agents can generate SQL or pipeline files, but they often guess table names, miss governance rules, or ignore lineage. ContextForge Agent was inspired by the Agent Hackathon because DataHub gives agents the missing context layer: schemas, ownership, lineage, quality checks, tags, and usage examples. We wanted to show that metadata-aware agents can produce practical data engineering artifacts instead of fragile drafts.

What it does

ContextForge Agent reads a DataHub-style catalog and builds a context bundle for a target dataset. It uses that context to generate a dbt model, dbt schema tests, an Airflow quality DAG, a DataHub write-back proposal, and an agent review note explaining which metadata was used.

The generated artifacts are designed to be reviewed like a pull request. They include real upstream table names, owner and domain metadata, lineage references, quality assertions, and governance notes such as avoiding upstream PII fields.

How we built it

We built the project as a standalone Python application with a small DataHub context adapter. The adapter mirrors the kinds of reads an agent would perform through DataHub's MCP Server or Agent Context Kit: search, get dataset details, inspect schema fields, and traverse upstream lineage.

The generator is deterministic so judges can inspect exactly how DataHub context affects the output. The repository includes sample metadata, generated artifacts, tests, and a command-line interface for listing datasets, inspecting a target, and generating new artifacts.

Challenges we ran into

The biggest challenge was making the demo realistic without requiring judges to connect to a private enterprise DataHub instance. We solved that by including a DataHub-like sample catalog while keeping the adapter boundary clear enough to replace with the real MCP Server or Agent Context Kit.

Another challenge was making the generated code useful rather than decorative. We focused on artifacts a data team would recognize: dbt SQL, dbt tests, an Airflow DAG, DataHub write-back JSON, and a review note that explains the reasoning.

Accomplishments that we're proud of

We are proud that ContextForge Agent demonstrates a full metadata-aware code generation workflow: read context, generate artifacts, preserve governance, and propose write-back to the catalog.

We are also proud that the generated examples are committed to the repository, so judges can evaluate output quality without running anything first.

What we learned

We learned that DataHub turns agentic data work from guessing into grounded generation. Schema fields, ownership, lineage, and quality assertions are not just documentation; they are inputs that can directly improve generated code.

We also learned that write-back matters. If an agent creates a model or pipeline but does not update the catalog, the next human or agent starts from incomplete context again.

What's next for ContextForge Agent

Next, we want to connect the adapter directly to the DataHub MCP Server and Agent Context Kit, then open a real pull request with generated dbt and Airflow artifacts. We also plan to add policy-aware generation, support for Dagster and Prefect, and a DataHub Skill that packages this workflow for other agents.

Built With

  • agent-context-kit
  • ai-agents
  • airflow
  • code-generation
  • data-engineering
  • datahub
  • dbt
  • governance
  • lineage
  • mcp
  • metadata
  • python
Share this project:

Updates

Submission history