ContextLease

Every AI-generated data change gets a DataHub-backed metadata lease. Before merge, ContextLease rechecks schemas, tags, lineage & owners - if context drifted, the PR is blocked.

Comment

Inspiration

AI agents are increasingly writing dbt models and data pipelines by reading metadata from catalogs like DataHub — schemas, tags, owners, lineage. That metadata is a snapshot in time. Between "the agent generated this" and "this actually merges," stewards can add a PII tag, remove an owner, or deprecate a table. The generated code still looks fine in the diff, but the assumptions it was built on are gone. Nothing in a normal code review catches that, because the risk isn't in the code — it's in the catalog.

I built ContextLease to close that gap: lease the DataHub context an AI artifact depended on, then deterministically re-check it before merge.

What it does

ContextLease snapshots DataHub metadata into a lease (contextlease.json), then runs a pure, deterministic verifier before merge — no LLM in the safety decision.

End-to-end flow:

  1. An MCP-enabled agent generates a dbt model using the DataHub MCP Server (mcp-server-datahub)
  2. ContextLease captures the exact metadata context in a lease (via DataHub GraphQL)
  3. A steward changes metadata in DataHub after the lease exists
  4. contextlease verify diffs leased vs. current snapshots and returns a specific verdict (unsafe_to_merge, regenerate_required, needs_owner_approval, etc.)
  5. Optional --write-back-datahub tags the dataset ContextLease-Blocked so the block is visible in DataHub itself, not just CI logs

Three validated verdict scenarios in the repo:

  • Sensitive tag added → unsafe_to_merge (live demo + write-back)
  • Owner removed → needs_owner_approval (examples 17–19)
  • Schema type change → regenerate_required (examples 14–15)

Agent generates. ContextLease governs. DataHub remembers.

How I built it

  • Python CLI (Typer + Pydantic) with a clean split: pure verifier (contextlease/verifier.py), I/O adapters (snapshot.py, datahub_graphql.py), and orchestration (cli.py)
  • DataHub MCP at generation time: contextlease agent-generate drives Claude Code against mcp-server-datahub; real tool calls are recorded in a generation trace with artifact SHA256
  • DataHub GraphQL at verify time: deterministic, repeatable lease refresh and diff — no network in the verifier itself
  • Sibling metadata merge: follows DataHub siblings so tags on either platform sibling are caught
  • Decision records + Markdown reports for audit; optional GraphQL write-back to DataHub
  • CI + branch protection: CI / test, CI / package, and ContextLease PR Gate / gate as required checks — a drifted lease produces a real red X on a PR, not just a log line
  • Reusable composite GitHub Action (.github/actions/contextlease-verify) — any repo can add uses: prabhakaran-jm/contextlease/.github/actions/contextlease-verify@v0.2 to a workflow today, not just this one
  • 91 tests, offline fixture path (init-demo) for judges with no DataHub required — plus a .devcontainer so judges can open the repo in a GitHub Codespace and run the offline demo with zero local setup
  • Apache 2.0 licensed, per the hackathon's submission requirements

Challenges I ran into

  • MCP vs GraphQL split: MCP gives the agent live context at generation; GraphQL gives ContextLease a deterministic, CI-grade re-check. Both are needed, for different reasons.
  • Truthfulness in examples: Early docs described some fixture reports incorrectly; I fixed them and added tests so example leases match documented verdicts.
  • Schema drift semantics: Had to extend the verifier beyond type changes to handle field add/remove and sensitive field tags without double-reporting.
  • Field-level tags hide in a different place: a PII tag added to a column via the DataHub UI lands in editableSchemaMetadata, not the schemaMetadata the dataset-level tag diff already covered — missing this meant column-level sensitivity changes were silently invisible to the verifier until I found and fixed it.
  • GitHub PR gate + rulesets: Required status check names had to match exactly what Actions reports (package, test, gate bound to GitHub Actions). Took several iterations to eliminate duplicate "Expected — Waiting for status to be reported" rows.
  • Live demo fragility: agent-generate needs Claude + uvx + DataHub and takes minutes — the video uses real runs with honest cuts, not fabricated output.

Accomplishments that I'm proud of

  • A novel, well-scoped problem with a clear "why not a linter" argument — the risk lives in metadata, not the diff
  • Real DataHub integration depth: MCP generation, GraphQL lease/refresh, sibling merge, and write-back to the catalog
  • Deterministic safety gate separated from I/O — same two snapshots always produce the same verdict
  • Honest signal handling: never fabricates a signal DataHub doesn't actually expose — quality/freshness reports unknown rather than guessing, and downstream lineage is documented as not-yet-captured rather than silently omitted
  • Rich demo artifacts: 20+ example files, live generation trace, red PR (drifted lease) and green PR (valid lease) for branch protection
  • Production-shaped CI: test, package, and PR gate workflows with a reusable composite action
  • Honest documentation: runbook, architecture docs, and a verifier hardened with schema add/remove rules validated by 91 tests

What I learned

  • DataHub's value for agents isn't just "read metadata" — contributing back (write-back tags, visible blocks in the catalog) makes governance actionable for the next human or agent
  • Determinism matters for merge-time safety: an LLM re-judging "is this safe?" would reintroduce the nondeterminism a lease is meant to eliminate
  • Submission quality is as important as code: judges may watch the video and skim the repo without running Docker; clear verdict tables, offline quickstart, and stable CI check names matter
  • GitHub rulesets bind to check-run names, not workflow display names — integration_id: 15368 (GitHub Actions) was the key to stable required checks

What's next for ContextLease

  • Formal GitHub Marketplace listing for the existing contextlease-verify composite action, so teams can discover and pin it without finding the repo first
  • Policy layer: org-specific severity overrides (e.g. column removed = hard block vs. warn)
  • Broader artifact types: Spark SQL, Airflow DAGs, ingestion configs — same lease pattern
  • Downstream lineage capture in leases (currently upstream only)
  • DataHub OSS contribution: docs or a small skill for the merge-time lease pattern
  • Optional hosted dashboard for decision records (static HTML prototype exists for judges)

Built With

  • 2.0
  • actions
  • ai
  • apache
  • ci/cd
  • claude
  • code
  • context
  • dataengineering
  • datahub
  • dbt
  • deterministic
  • github
  • governance
  • graphql
  • mcp
  • metadata
  • model
  • protocol
  • pydantic
  • python
  • server
  • typer
  • verification
Share this project:

Updates