ContextTwin CI

From a breaking data change to governed, merge-ready code — DataHub finds the hidden consumer, ContextTwin repairs it, Replay proves it, and the mapped owner approves it.

Comment

Inspiration

A schema-changing pull request can compile, pass every local dbt test, and still break a dashboard, a pipeline, or an operational query owned by another team. The reviewer sees green checks. The consumer finds out in the morning.

The missing input is not in the repository. It is organizational context — real schemas, downstream lineage, stored queries, ownership, certification. DataHub already holds all of it.

So the question we started from was not "can an agent detect this?" Detection is solved; Recce and Datafold do it well. The question was: once the context is known, can an agent take the whole governed action? Generate the repair, prove it against the real consumer query, route approval to the actual owner, and write the accepted decision back so the organization keeps the knowledge.

What it does

ContextTwin CI runs on a schema-changing pull request and:

  1. Captures the exact GitHub base and head worktrees and compares their dbt manifests.
  2. Reads external schema, lineage, query, ownership and certification evidence through the DataHub MCP server. Truncated or ambiguous context is refused, not guessed.
  3. Makes a deterministic SAFE / REPAIR_REQUIRED / REFUSE decision from versioned policy-as-code loaded from the PR base SHA — so a pull request cannot weaken the policy that judges it.
  4. Accepts only a typed RepairIntent from the model, then deterministically compiles the bounded dbt, SQL, test and migration artifacts.
  5. Validates them with dbt and DuckDB inside a non-root, networkless Docker sandbox.
  6. Runs ContextTwin Replay: the affected consumer's real query is executed three times — baseline, breaking change, repaired.
  7. Requires baseline PASS → changed FAIL → repaired PASS, with the repaired column names, types, row count and normalized SHA-256 result hash matching the baseline exactly.
  8. Accepts approval only from the mapped numeric GitHub user ID, for the exact 16-character Change ID, against the immutable head SHA.
  9. Writes the Decision to DataHub, reads it back to verify it, anchors the full proof digest on a GitHub commit status, and only then publishes MERGE_READY.

How we built it

The core design decision was to give the model less authority, not a better prompt.

AI may return a typed RepairIntent — an enum action, URNs, and identifier names. It never emits SQL, Jinja, or file paths. A deterministic SQLGlot compiler turns that intent into the actual artifacts, and it verifies the intent's consumer set exactly equals the evidence's affected set before compiling anything. Everything downstream — policy, validation, Replay, approval, write-back — is deterministic and testable.

Around that we built a tamper-evident proof chain: RFC 8785 canonical JSON with domain-separated SHA-256 digests over the manifest projections, policy decision, DataHub evidence graph, repair intent, artifact bundle, validation result and Replay evidence. Lifecycle events bind the previous event's digest. contexttwin verify recomputes all of it rather than trusting the serialized labels.

Isolation is layered. dbt runs in a Docker container with --network none, --read-only, --cap-drop ALL, no-new-privileges, PID/memory/CPU limits and a digest-pinned base image. Each consumer query runs in a separate read-only worker process with DuckDB external access disabled, extension autoloading off, lock_configuration frozen, a 10-second timeout and a row ceiling — and it returns only schemas, counts and hashes. Raw consumer rows never leave the worker and never enter the proof.

Challenges we ran into

The hardest problem was not code generation. It was proving that every external action referred to the same immutable change.

We had to handle paginated GitHub API output across CLI versions, bind DataHub queries to direct consumers without accepting ambiguous lineage, work with DataHub MCP document APIs that expose search excerpts separately from document metadata, and close a time-of-check/time-of-use gap by binding validation to captured worktrees instead of mutable branch paths.

A Linux-only bind-mount ownership bug surfaced only in public CI: a non-root UID could not create dbt's target/, and afterwards the runner could not remove the directory that UID owned. Fixing it properly meant moving manifest parsing into an isolated writable copy and running the container as the calling POSIX UID when that UID is non-root.

Each of those constraints ended up becoming a product feature: fail-closed evidence retrieval, exact-SHA approval, read-back verification, and a digest-linked state history.

Accomplishments that we're proud of

  • A real REPAIR_REQUIRED → APPROVAL_PENDING → APPROVED → WRITTEN_BACK → MERGE_READY chain against a live pull request and a live DataHub instance.
  • Result-level green-red-green Replay that never stores a raw consumer row.
  • A proof that rejects fixtures, forks, stale SHAs, stale approvals, changed evidence, unmapped owners, truncated lineage, failed replay, failed write-back, or a missing digest anchor.
  • 238 tests at 75.28% statement coverage, including E2E tests that invoke real networkless Docker/dbt/DuckDB validation, plus adversarial tests for proof tampering, path traversal, symlinks, SQL injection shapes, replay query boundaries and secret redaction.
  • Three secret-free committed scenarios and a public LIVE evidence index, so the project can be evaluated without operating the full stack.

Honest scope

The Finance catalog is transparently seeded and synthetic. It is a reproducible scenario, not a claim about an undisclosed production incident. What is live is the integration: DataHub MCP discovery, GitHub identity checks, Replay execution, approval binding, the external write-back and its read-back verification.

Replay is deliberately narrow for this hackathon: DuckDB only, one read-only single-table SELECT per consumer, with joins, CTEs, subqueries, unions, parameters, table/file/network functions and nondeterministic functions all rejected rather than guessed at.

What we learned

Metadata becomes far more valuable when it is part of a closed engineering loop. Reading lineage is useful. Using lineage to generate a bounded repair, prove that repair against the real consumer query, route approval to the actual owner, and write the accepted decision back — that creates reusable organizational knowledge.

We also learned that trustworthy agents need smaller authority, not broader prompts. Typed intent surrounded by deterministic compilers and explicit state gates produced a system that was easier to test, easier to explain, and easier to trust than any amount of prompt engineering would have been.

What's next for ContextTwin CI

  • Expand Replay beyond the single-table DuckDB scope while keeping fail-closed query validation.
  • Add warehouse and orchestration artifact compilers beyond dbt.
  • Support hosted DataHub environments so the LIVE demo can be reached remotely.
  • Push CLI and live-adapter coverage past 80%.
  • Contribute reusable integration learnings upstream where they fit DataHub's contribution scope.

Try it in 30 seconds

No DataHub server and no provider key required:

cd app
uv sync --frozen
uv run contexttwin verify --proof ../examples/repair-required/migration-proof.json
uv run contexttwin demo repair-required

The committed fixture is deliberately DEMO_ONLY — it proves reproducibility without pretending to authorize external writes. The separate public LIVE evidence points to the exact pull request, owner approval, full-digest status anchor, verified DataHub Decision write-back, and seven lifecycle records from a real end-to-end run.

Built With

  • datahub
  • dbt
  • docker
  • duckdb
  • github-actions
  • openai
  • pydantic
  • python
  • sqlglot
  • typer
Share this project:

Updates