FixtureForge

Compile governed DataHub metadata into deterministic, source-row-free test assets with independent proof, a caught negative control, and verified MCP writeback.

Comment

Inspiration

Data teams routinely choose between copying production rows and inheriting privacy risk, or hand-writing fixtures that drift from schemas and relationships. DataHub already contains the governed specification developers need, but turning that metadata into code still takes repetitive work.

What it does

FixtureForge is a bounded autonomous metadata-to-code agent. From one natural-language goal it searches DataHub through the official MCP Server, expands the selected lineage graph, and inspects schema, keys, lineage, ownership, tags, glossary terms, and governance context. It then emits linked CSV and Parquet fixtures, dbt tests, typed Python accessors, and a provenance manifest without reading production source rows.

The agent independently validates every constraint with DuckDB, injects a broken foreign key to prove the validator can reject bad output, rebuilds to prove byte identity, and produces a reviewable Git change. With explicit approval, it writes a structured DataHub Context Document containing the evidence fingerprint and Git receipt, then verifies the write through a read-after-write check.

How we built it

FixtureForge uses DataHub OSS v1.6.0, the official mcp-server-datahub package pinned at 0.6.0, the MCP Python SDK, Pydantic, DuckDB, Typer, uv, pytest, Ruff, and mypy. The agent has observable plan, discover, inspect, generate, verify, rebuild, deliver, and writeback states. The compiler topologically orders foreign-key dependencies and scopes pseudo-random generators by seed, dataset, and field.

Challenges

The hardest part was making each claim auditable. Search can overmatch, so the agent applies an exact namespace boundary when the goal names one. The MCP transport is pinned and allowlisted. Git delivery refuses dirty repositories and unsafe destinations. Source-row-free is explicitly not promoted into an anonymization, compliance, production-readiness, adoption, or time-savings claim.

Accomplishments

  • real goal-to-search-to-lineage discovery on DataHub OSS;
  • schema, lineage, ownership, tags, glossary terms, and governance context read through the official MCP Server;
  • deterministic CSV, Parquet, dbt, and typed factory outputs;
  • 30 live support-domain checks, a caught negative control, and a byte-identical rebuild;
  • agent-generated delivery PR and mergeable ownership/adversarial-evidence PR;
  • structured DataHub Context Document with full-fingerprint readback;
  • 27 automated tests plus strict type and lint checks;
  • 6/6 compatible adversarial schema changes passed on the first attempt; one invalid relationship was refused before generation;
  • local compatible-scenario latency median 2.8801 seconds and p95 4.4119 seconds;
  • upstream DataHub fixture-generation Skill PR;
  • 2:07 live evidence video; and
  • public Apache-2.0 source with a reproducible recorded-MCP judge path.

What we learned

Metadata is executable context, but it is not automatically correct. Agents need visible scope decisions, deterministic evidence, independent verification, human approval at mutation boundaries, and receipts that downstream reviewers can inspect.

What is next

Add DataHub Assertions and structured properties, support richer multi-hop graph selection, and validate the workflow with external data teams. Current evidence proves local operation, public CI, and reviewable artifacts, not production adoption or time savings.

Built with

DataHub OSS, official DataHub MCP Server, Python, MCP SDK, Pydantic, DuckDB, Typer, uv, pytest, Hypothesis, Ruff, mypy, Git, and GitHub Actions.

Try it

AI assistance disclosure

OpenAI Codex substantially assisted with competition research, architecture, implementation, testing, documentation, evidence review, and video production. The entrant directed the scope, authorized external actions, and remains responsible for the project and its claims.

Built With

Share this project:

Updates