• 

  iGraph

iGraph: From DataHub Lineage to Enforceable Agent Authority

Inspiration

Autonomous coding and data agents are becoming capable of writing SQL, modifying dbt models, changing schemas, and repairing pipelines.

The problem is that an agent may understand the file or query it is editing without understanding the organizational consequences of that change.

A request such as:

Rename customer_id to account_id.

may look local, but the column could feed downstream transformations, dashboards, ML features, reports, and production workflows owned by different teams.

That dependency information already exists in DataHub. What is missing is an execution boundary that converts this context into a decision about what an agent is actually allowed to do.

That is why I built iGraph.

What iGraph Does

iGraph is a context-aware execution control plane for autonomous data agents.

It converts DataHub metadata into bounded, verifiable authority:

Change request
    ↓
Impact Graph
    ↓
Risk and policy evaluation
    ↓
Signed Impact Pact
    ↓
Policy Enforcement Point
    ↓
Executor or denial
    ↓
Post-change verification
    ↓
Change Receipt

The Impact Graph captures the consequences of a proposed change: downstream lineage, schemas, owners, domains, tags, glossary terms, quality assertions, query usage, dashboards, and ML dependencies.

The graph is compiled into an Impact Pact. The Pact binds:

• the original request;
• the authoritative DataHub context;
• a canonical context fingerprint;
• the risk assessment;
• allowed and blocked actions;
• execution targets;
• artifact hashes;
• required validations;
• approval requirements;
• expiration time;
• postconditions.

The agent can propose a change, but it cannot silently widen the authority granted by the Pact.

Immediately before an external side effect, iGraph’s policy enforcement point checks the signed Pact again. It validates the requested action, target, artifact hash, approval state, expiry, and invocation limit.

The result is explicit:

• allow;
• deny;
• require approval;
• require migration review;
• context unavailable;
• Pact stale.

After execution, iGraph re-reads the context and evaluates the Pact’s postconditions. It then produces a machine-readable Change Receipt containing the decision, execution evidence, validation results, artifact hashes, and optional DataHub write-back information.

How I Built It

I built the control plane with FastAPI, Pydantic models, and a deterministic impact engine.

The DataHub adapter uses the DataHub SDK together with the DataHub Agent Context Kit to retrieve:

• entity metadata;
• multi-hop downstream lineage;
• schema fields;
• ownership and domains;
• tags and glossary terms;
• assertions and quality status;
• query history;
• context documents.

The impact engine normalizes this information into an Impact Graph, calculates a risk score, generates reviewable artifacts such as an impact report, migration SQL, and regression checks, and compiles the result into a signed Impact Pact.

Pacts are HMAC-signed, expire after a short time window, and contain a canonical SHA-256 context fingerprint. This means that changing the request, context, policy, artifact, target, or execution scope invalidates the authority object.

The enforcement point also prevents replayed executions and refuses to invoke the executor when a target or artifact falls outside the Pact scope.

The frontend is a control-room interface for inspecting the graph, risk, Pact, validations, execution decision, and receipt. The backend is deployed as a FastAPI service, while the frontend can be hosted separately.

The project supports both explicit demo mode and live DataHub mode. In live mode, missing or incomplete DataHub context fails closed with context_unavailable; it never silently falls back to demo data.

What I Learned

The most important lesson was that lineage is not the same thing as authority.

A catalog can tell an agent what depends on a dataset. It does not automatically tell the agent what it may change, which actions require approval, or whether a deployment should be blocked.

I also learned that signing a policy object is not enough. A secure execution boundary must validate the Pact immediately before the side effect, bind the action to an approved artifact and target, prevent replay, and verify the result afterward.

The same request and executor can receive different authority when the surrounding organizational context changes. That makes context an active security input, not merely background information.

Challenges

The hardest design problem was deciding what to do when DataHub context is missing, incomplete, truncated, or temporarily unavailable.

A permissive system could continue with a best-effort answer, but that would create the exact failure mode iGraph is intended to prevent. I chose fail-closed semantics: if the system cannot establish authoritative context, it cannot authorize the side effect.

Another challenge was preserving the boundary between planning and execution. The agent is allowed to propose a migration and generate artifacts, but the final executor must only receive actions that remain inside the signed Pact.

Finally, running the complete DataHub stack requires several stateful services, including GMS, Kafka, MySQL, and OpenSearch. That made deployment and persistence more difficult than deploying a normal stateless API. iGraph therefore keeps DataHub access server-side and makes the demo/live boundary explicit.

What Comes Next

The next step is connecting the live adapter to a reachable DataHub OSS instance, loading a permitted sample graph such as showcase-ecommerce, and demonstrating the complete lifecycle with real lineage:

DataHub context → Impact Graph → Impact Pact → guarded execution → verification → Change Receipt

The long-term goal is to make iGraph a reusable authority layer for any autonomous system that wants to change data infrastructure without losing organizational context, policy control, or evidence of what actually happened.

Built With

• autonomous-agents
• css
• data-governance
• data-infrastructure
• data-lineage
• datahub
• datahub-agent-context-kit
• fastapi
• hmac
• html
• javascript
• metadata
• policy-as-code
• pydantic
• python
• render
• rest-api
• security
• sha-256
• vercel