ContextForge

Inspiration Every data team has the same problem: AI code generators produce data pipeline code that looks right but breaks in production. Column names are wrong, JOIN keys are guessed, and governance policies are ignored. We've all seen a ChatGPT-generated dbt model fail because it hallucinated a column name that doesn't exist.

The root cause is always the same — the AI has no context about your actual data. DataHub solves this. It's an open-source metadata platform that already knows every column name, every foreign key, every lineage edge, and every governance tag in your data stack.

ContextForge was born from a simple question: What if an AI agent could read DataHub before writing a single line of code?

What It Does ContextForge is a web application that generates production-ready data pipeline code by querying DataHub's context graph through the MCP Server and Agent Context Kit. It supports four output types:

dbt Models — SQL transformations with correct {{ source() }} calls, proper JOIN keys from FK metadata, and PII column exclusion based on governance tags Airflow DAGs — Python DAGs with extraction tasks per source dataset, Snowflake transformations, data quality validation, and owner-based alert routing SQL Transformations — Platform-aware CREATE TABLE AS statements with CTEs and JOINs derived from DataHub's lineage graph Ingestion Scripts — Python scripts with PII hashing, schema validation from DataHub, and lineage write-back The key differentiator is the Agent Reasoning Panel — a transparent view into every DataHub query the agent makes, what it found, and how it used that information. No black box.

How We Built It ContextForge is built with Next.js 16, TypeScript, Tailwind CSS 4, and shadcn/ui. The architecture has three layers:

DataHub Integration Layer (src/lib/datahub-mock.ts) — Simulates DataHub's MCP Server API with realistic e-commerce data (8 datasets, 86 columns, 3 platforms). In production, this would be a direct MCP Server connection. Code Generation Engine (src/lib/code-generator.ts) — A multi-step agent that queries schema metadata, traces lineage for JOIN keys, checks governance tags for PII, and looks up ownership before generating code. Each step is recorded as a reasoning step shown to the user. Interactive Frontend (src/app/page.tsx) — A single-page application with 5 views: Dashboard, Metadata Explorer, Code Generator, Lineage Graph, and Generated Projects. The code generator doesn't use a generic LLM to produce code. Instead, it uses deterministic, template-based generation that's guaranteed to produce syntactically correct output because it reads exact column names and types from DataHub.

Challenges We Faced Jinja in JavaScript — dbt models use {{ }} delimiters that conflict with JavaScript template literals. We solved this by building strings through concatenation, ensuring the generated output contains correct Jinja syntax. Lineage-Aware JOINs — Determining correct JOIN conditions requires tracing FK relationships through DataHub's lineage graph. We built a relation-finding algorithm that works bidirectionally through the schema relations. Governance-Aware Generation — PII columns must be excluded from output, not just masked. The agent reads DataHub governance tags to automatically filter PII columns from SELECT statements and add PII hashing in ingestion scripts. What We Learned DataHub's metadata graph is incredibly rich for code generation — FK relationships alone solve the hardest part of generating correct JOINs Transparent agent reasoning (showing each MCP query) builds more trust than a magic black-box generator The Agent Context Kit provides exactly the right abstraction level for an agent that needs to understand data context DataHub Technologies Used DataHub MCP Server — Primary interface for querying schemas, lineage, tags, and ownership Agent Context Kit — Used for lineage traversal and platform detection DataHub Skills — Used for lineage write-back (generated code creates new lineage edges) What's Next Connect to a real DataHub GMS endpoint via the MCP Server Add support for Spark and dbt-core DAG generation Implement a PR creation workflow that sends generated code to Git with DataHub-linked metadata Add a natural language interface ("Create a dbt model joining orders and customers") Support for DataHub's ML metadata to generate feature engineering pipelines Built With datahub, nextjs, typescript, tailwindcss, shadcn-ui, prisma, mcp-server, agent-context-kit

Built With

• next.js
• tailwind
• typescript