Graph Traffic Control

File locks can't see a semantic collision. Graph Traffic Control gives autonomous data agents a DataHub-powered two-phase commit that can.

Comment

Inspiration

Coding agents coordinate files, branches, worktrees, and task queues, but those mechanisms cannot see a semantic dependency in a data graph. One agent can rename a column while another publishes a metric that reads it through lineage. The proposals touch different files and declare no shared asset, so a file-level coordinator says they are safe even though the combined result is broken.

Serializing every agent would avoid the collision and also block unrelated work. I built Graph Traffic Control to answer the harder question: what does a proposed change actually touch, and can DataHub's governed graph become the coordination boundary?

What it does

Agents submit structured proposals containing identity, intent, explicit read and write URNs, expected versions, an executable action, validation, lease duration, and risk. The coordinator:

  1. expands each proposal through bounded DataHub lineage, schemas, ownership, tags, and domain;
  2. evaluates a documented conflict matrix, including a lineage-mediated conflict between two proposals that declare no overlapping URN;
  3. reports the shortest directed lineage path that proves the conflict;
  4. prepares safe work with expiring leases and a token fingerprinting its graph context;
  5. allows an unrelated proposal to commit instead of queueing it behind the revenue branch;
  6. routes high-blast-radius work for human approval;
  7. rereads the graph immediately before commit and fails closed on drift;
  8. executes the artifact change, validates it, and verifies it by reading the file back; and
  9. records every state transition and receipt in an append-only audit log.

The public fixed scenario submits four proposals and finishes with three verified commits and one fail-closed abort. The key result is not an ordinary shared-file conflict: the A/B collision has zero declared asset overlap and is visible only through lineage.

How I use DataHub

The DataHub MCP Server supplies get_entities, get_lineage, and list_schema_fields context for the allocated traffic.* graph. Lineage produces the hidden A/B conflict, criticality determines which work needs approval, and the schema, ownership, tag, and domain fingerprint determines whether a prepared proposal is still safe to commit.

The public one-button scenario uses an equivalent isolated in-process fixture so repeated anonymous runs cannot mutate the shared catalog. Separately, the live application provider and readiness gate verified all nine allocated entities and seven lineage edges, then performed a supported description write, immediate reread, and exact restoration. The story keeps those two evidence boundaries explicit.

How I built it

The coordinator uses Python, FastAPI, strict Pydantic models, NetworkX, SQLite, expiring leases, and a nine-state transaction machine. A small HTML, CSS, and JavaScript console renders the graph, proposal state, conflict evidence, commit signals, and audit trail.

COMMITTED means proved rather than attempted. Mutation, mutation reread, validation, writeback verification, restoration, rollback, and receipts are independent signals. The deterministic coordinator owns conflict and commit decisions; no model output participates in them.

Challenges I faced

The first challenge was avoiding the dangerous interpretation of an MCP failure as an empty graph. An empty graph looks conflict-free, so unknown response shapes, missing context, or tool errors now abort the read instead of becoming “safe.”

I also had to make prepare meaningful. A lease alone is insufficient if lineage or schema changes between review and execution, so each prepared token includes a graph fingerprint and the coordinator rereads that state immediately before commit.

Live verification found a real DataHub compatibility mismatch: the default description operation SET was rejected while replace was accepted. I corrected the configuration, performed the reversible write/reread/restore proof, and preserved that compatibility decision in the project record.

What I learned

Concurrency safety depends on the level where conflicts actually exist. File locks solve file collisions; they cannot solve semantic collisions. I also learned that an error returning “no context” is more dangerous than an obvious crash because it can look like permission to proceed. Finally, a transaction label is only credible when the system separately proves mutation, validation, reread, rollback, and audit evidence.

What I am proud of

  • The defining conflict has zero declared overlap and is revealed only by DataHub lineage.
  • Unrelated work proceeds instead of being globally serialized.
  • Expiring leases, graph fingerprints, approval, and drift checks are visible in one workflow.
  • Three proposals commit and one aborts safely in the deterministic public scenario.
  • The live DataHub write path was verified and restored without changing sibling project state.

Limitations and what's next

Graph Traffic Control does not provide distributed ACID transactions across external platforms, solve multi-agent coordination in general, or guarantee exactly-once effects. The public scenario uses a fixed fixture and local artifact actions. Next I would add authenticated agent identities, shared lease storage, connectors for orchestration platforms, and policy-assisted conflict resolution that remains subordinate to deterministic safety checks.

Built With

Share this project:

Updates