Datahub Causality Agent

HTTP 200 can hide a broken business path. This agent uses DataHub schema and lineage to find the risky workflow, prove the route changed, and write verified evidence back to the affected asset.

Comment

Inspiration

Many production failures do not look like failures at the HTTP layer. An API can return 200 OK while silently taking a different business path: skipping a status check, falling into a retry queue, returning a field with a different meaning, or bypassing a downstream operation entirely. Conventional uptime monitoring sees a successful request. A metadata catalog sees the affected asset and its relationships. Neither one, by itself, proves what the changed system actually did.

That gap is the reason for DataHub Causality Agent. The project treats DataHub as the context graph for deciding what deserves runtime validation, then compares the same workflow against a baseline and a changed environment. The output is not a generic alert or a dashboard metric. It is a verified causal finding, attached to the affected DataHub asset so the next engineer or agent inherits the evidence.

The motivating example is a checkout API. The baseline returns:

{"id":"order-1","status":"paid"}

The changed API returns:

{"id":"order-1","state":"paid"}

Both requests return HTTP 200. The semantic difference causes the workflow to take a different route: the baseline proceeds to status polling, while the changed environment goes to the retry queue. This is the kind of regression that can remain invisible until a customer, a downstream service, or an operator discovers its consequences.

What it does

DataHub Causality Agent is an evidence-gated agentic control loop for runtime behavior validation:

  1. It reads the changed asset from DataHub through the official DataHub MCP server. The proof verifies successful get_entities, list_schema_fields, and get_lineage calls rather than claiming MCP usage from configuration alone.
  2. It normalizes the returned schema, ownership/criticality signals, and downstream lineage into a stable planning model. In the live proof, DataHub exposes the checkout schema as id,state, marks it as tier-one criticality, and identifies the downstream status and retry assets.
  3. It ranks executable workflows using that graph context. The selected checkout_paid_path scores 70 because it is in the downstream closure, has high criticality, and depends on the missing routing field status. The impacted but less specific status_observer_path scores 10. The workflow catalog defines executable HTTP surfaces; DataHub supplies the facts that make the selection accountable.
  4. It generates one workflow scenario and runs it with identical inputs against a baseline API and a changed API. The deterministic demo services are intentionally small so the evidence is visible: the only meaningful change is the checkout response and the resulting route.
  5. It parses structured route evidence instead of trusting status codes. The first divergence is the checkout step: /orders/order-1 in the baseline and /orders/queue/retry in the changed environment, with HTTP 200 on both divergent requests.
  6. It gates write-back on evidence. A native DataHub Incident is written only when the comparison reports DRIFT, names a root divergent step, and contains structured baseline and changed routing sides.
  7. It reads the exact Incident back through the affected dataset's paginated dataset.incidents connection. The proof checks the Incident URN, ACTIVE status, finding ID, evidence fingerprint, changed asset, impacted assets, workflow, root step, and routing difference.
  8. It records compatibility honestly. The released MCP server's generic Incident entity lookup is probed separately. When that resolver cannot return the native Incident on the pinned released stack, the result is recorded as unsupported_by_released_stack; it is not presented as a false success. The native DataHub readback remains authoritative.

The project is intentionally a control loop rather than a frontend. Its interface is the evidence it produces: the planning decision, the causal trace difference, and the native DataHub Incident. A judge can see the complete proof from a terminal and the DataHub asset page without relying on a simulated chat conversation or a decorative dashboard.

How we built it

The implementation is organized around a small number of deep boundaries:

official DataHub MCP context
        -> normalized schema/criticality/lineage model
        -> metadata-driven workflow ranking
        -> generated executable scenario
        -> identical baseline/changed execution
        -> structured causal route diff
        -> native DataHub Incident write-back
        -> exact dataset.incidents read-after-write

agent/datahub_mcp_client.py owns the official MCP stdio boundary. It controls the child process environment, required-tool checks, timeouts, structured response parsing, schema/lineage pagination, result hashing, and a sanitized tool-call ledger. The live E2E proof refuses to continue unless the context transport is official-datahub-mcp and all required context calls are verified.

agent/datahub_context_source.py converts DataHub responses into the planner's stable model. It normalizes schema fields, tags, glossary/domain signals, criticality, ownership, composite URNs, and downstream relationships. It also compares the live schema with the executable workflow's asserted fields so a missing routing field becomes explicit planning evidence instead of an opaque score.

agent/agent.py performs the agent loop. It computes downstream closure, ranks candidate workflows, generates the scenario, runs the same scenario against the two environments, and parses the machine-readable diff. The selected workflow is therefore a consequence of DataHub context plus the workflow catalog—not a hard-coded if checkout then run checkout branch.

agent/datahub_e2e_proof.py provides the reproducible proof harness. It seeds a local DataHub OSS instance, waits until the schema, criticality, and lineage context converges, acquires that context through MCP, starts deterministic baseline and changed API processes, runs the agent, checks the route evidence, probes MCP Incident compatibility, and replays the same finding.

agent/datahub_writeback.py owns the correctness boundary for the resulting finding. It refuses writes without verified routing drift, creates or updates a native DataHub Incident, waits for read-after-write convergence, paginates the asset's Incident connection, and verifies the complete evidence payload. The same finding ID and Incident URN are reused on serial replay, with exactly one matching Incident in the proof.

The repository also includes a fail-closed negative control. It runs the same workflow against identical API variants and requires NO_DRIFT, no Incident URN, zero matching Incidents, and suppressed write-back. This demonstrates that the system does not create an alert merely because a workflow was executed.

The runtime uses Python 3.12, DataHub OSS, the official DataHub MCP server, GraphQL for the authoritative native Incident readback, Docker/Compose for the local DataHub deployment, and a pinned external execution engine for the baseline/changed trace comparison. The repository is Apache 2.0 licensed. The external execution engine, ACE v0.1.7, is MIT licensed, predates the hackathon, and is used unmodified; the DataHub integration, planner, proof harness, write-back path, compatibility classification, tests, documentation, and sample artifacts are the new work in this submission.

Architecture and proof visuals

The three proof-oriented diagrams are uploaded separately in Devpost's Project Media gallery. Read them in this order:

  1. architecture-control-loop.png — system ownership and data flow.
  2. evidence-gate.png — the fail-closed write gate.
  3. proof-board.png — the concrete HTTP 200 / route-divergence result.

The gallery is deliberately part of the evidence: the architecture shows which component owns each decision, the gate shows why an Incident is earned rather than inferred, and the proof board shows the verified DataHub write-back.

Challenges we ran into

The first challenge was separating metadata risk from runtime proof. A schema change and a lineage edge can tell us which workflows are plausible victims, but they cannot prove that a deployed service actually changed behavior. We solved that by making the graph select and scope the test, then requiring a baseline/changed execution with preserved route evidence before any Incident can be written.

The second challenge was the difference between transport success and business success. The demo's two checkout requests both return 200, so a status-only check would pass. The generated scenario asserts the business response and captures the route taken after that assertion. The proof reports the first structured route divergence rather than reducing the result to an HTTP status.

The third challenge was proving DataHub usage end to end. It is easy for a demo to mention a catalog while using a local fixture for its real decision. This project keeps the executable workflow catalog separate, but obtains the changed asset's schema, criticality, and lineage through the official MCP server. The live proof records the MCP tool ledger, transport identity, context snapshot, and planning evidence, and it fails if the required MCP calls are missing.

The fourth challenge was writing a finding back safely. A mutation response is not proof that the intended Incident is visible, complete, or unique. The write-back path reads the native Incident from DataHub, verifies the exact finding identity and evidence fingerprint, and checks the ACTIVE status and resource association. It also refuses to hide duplicate exact matches.

The fifth challenge was compatibility with the released MCP stack. Native Incident creation and the dataset Incident connection work, while the generic MCP Incident entity resolver is not available on the pinned released versions. Rather than silently substituting a different read path or claiming unsupported capability, the project reports the limitation explicitly and keeps the native DataHub connection as the authoritative readback.

Finally, idempotency needed a precise boundary. The proof demonstrates serial replay idempotence: the same verified finding reuses one finding ID and one Incident URN. DataHub's mutation does not provide a caller-selected Incident URN or an atomic source-identity constraint, so this project does not claim concurrent exactly-once semantics. That limitation is part of the result, not hidden behind the word "idempotent."

Accomplishments that we're proud of

  • A complete DataHub-to-action-to-DataHub loop using the official MCP server.
  • A concrete silent-failure proof where both APIs return HTTP 200, but the business route changes from polling to retry.
  • Metadata-driven workflow selection with visible reasons, ranking, and impacted-asset closure.
  • Structured causal evidence that identifies the first divergent step and both runtime routes.
  • Native DataHub Incident write-back gated on verified DRIFT evidence.
  • Exact paginated native Incident read-after-write verification.
  • Serial replay proof that reuses one finding identity and one Incident URN.
  • A negative control proving that unchanged behavior produces NO_DRIFT and no write-back.
  • A released-stack MCP compatibility report that distinguishes supported native behavior from an unavailable generic resolver.
  • Credential-redacted runtime artifacts, an MCP tool ledger, a hash manifest, portable sample output, and reproducible setup/check commands.
  • A test and verification path that currently passes 44 unit tests, ACE scenario validation, the negative control, sample proof verification, and dependency checks through make check.

What we learned

DataHub context becomes significantly more useful when it controls an action, not just a lookup. Lineage and criticality can narrow a potentially enormous runtime test space to the workflow most worth proving. But context is a hypothesis: only execution against comparable environments can establish what actually changed.

We also learned that evidence gates should be designed around the artifact the next person needs. In this case, that artifact is not a model confidence score. It is a causal route difference tied to a DataHub asset, with enough identity and readback checks that another engineer can distinguish a real finding from a successful API call or a stale mutation response.

The compatibility work reinforced a broader lesson: integrations are not simply "working" or "broken." A released stack may support native Incident mutation and asset-scoped readback while not supporting generic MCP Incident retrieval. Recording that boundary makes the system more trustworthy and gives upstream maintainers a precise improvement target.

What's next for DataHub Causality Agent

The next production step is a concurrency-safe source identity for findings. The current serial replay contract is useful for a single agent, but a multi-worker deployment needs an atomic deduplication primitive or a caller-selectable idempotency key at the DataHub mutation boundary.

We would also expand the workflow catalog beyond the checkout example, add adapters for real staging environments with explicit safety controls, and make the planning model consume richer DataHub governance signals. Once the released MCP stack exposes a reliable native Incident resolver, the compatibility probe can become a second verified read path rather than a documented limitation.

The core thesis would stay the same: use the metadata graph to decide what matters, execute the smallest comparable proof, and write only evidence that survives read-after-write verification back into the graph.

Built during the hackathon

All DataHub integration, MCP context acquisition, workflow selection, scenario generation, deterministic E2E proof harness, native Incident write-back, read-after-write verification, serial replay idempotence, compatibility classification, tests, docs, and sample artifacts in this repository.

Prior work

ACE v0.1.7 is disclosed prior work and is used unmodified as a pinned MIT-licensed execution dependency. Everything that makes this a DataHub agent—the official MCP boundary, metadata-driven planning, scenario generation, closed-loop execution, native Incident write-back, exact readback, replay idempotence, and proof harness—was built during the submission period. See PRIOR_WORK.md.

How to run

make setup
make datahub-up
make demo
make demo-proof
make verify-sample

Built With

Share this project:

Updates