PDX DataHub Gate

Inspiration

Most “agents for metadata” stop at advice: they summarize a dataset, suggest a tag, and leave the human to click around in a UI. That is not real work on the catalog.

Real work in this context means the agent (or the tool it drives) actually changes DataHub state, with evidence of what was read, what was proposed, and why the write was allowed. We wanted that mutation path to be safe enough for demos and honest enough for production thinking: no silent add_tags, no skipped review, no fake “I understood the document.”

So we built PDX DataHub Gate, an embeddable approval choke point. An agent proposes a change; the gate loads live (or fixture) context, builds a change_package_v1, runs PDX verify/approval, and only then writes back through DataHub MCP. Upstream ProDocuX and PDX Artifact Engine stay existing repositories we depend on; this project owns the DataHub-facing agent work loop.

What it does

PDX DataHub Gate lets agents do catalog work, not just talk about it:

1. Agent / host submits intent - for example “add pdx.change_reviewed to this dataset,” optional note, optional file evidence.
2. Gate does real reads - DataHub context via fixture, GraphQL, or MCP (get_entities and related tools).
3. Gate structures the job - builds change_package_v1, verifies, and requires approval (human sign-off, or explicit auto-approve for local demos).
4. Gate does real writes - only after approval, MCP add_tags (confirmed in the DataHub UI on SampleHiveDataset).
5. Host gets a machine response - GateResult.status / host_action so the agent loop can refresh UI, ask a human, await approval, or stop, not blindly retry mutations.

Optional ProDocuX profiles attach deterministic file evidence to the same package. Notes may stay in artifacts when MCP does not persist them - we disclose that instead of claiming a full institutional-note product.

What this hackathon project is (and is not)

In one line: agents propose; the gate forces verify/approve; MCP performs the real DataHub side effect.

How we built it

Existing repos (dependencies)

• ProDocuX - Apache-2.0 Kernel for deterministic intake / structure checks over HTTP /v1. Used when evidence mode is live (or stub for offline demos).
• PDX Artifact Engine - model-optional orchestration: execution plans, tool registry, verification, approval, and run/artifact manifests. The gate registers DataHub tools on this runtime and validates plans with Core schemas.

We treat both as published, version-pinned libraries, not as features invented inside the hackathon vertical.

What we built so agents can do real work

• DataHub read/write transports (fixture, GraphQL, MCP stdio to mcp-server-datahub)
• change_package_v1 builder and domain verifiers (ownership review, disclosure, actions)
• Approval-gated write-back (add_tags with current MCP tag_urns / entity_urns)
• Optional agent_run loop (OpenAI / Gemini) that still cannot bypass the gate
• Host API run_change_gate → GateResult and a fake host that branches like a real system

from pdx_datahub_gate import run_change_gate

result = run_change_gate(
    intent=intent,
    output_dir="runs/demo",
    auto_approve=True,
    datahub_mode="mcp",  # or fixture | live
)
# Agent/host: refresh_ui | ask_human | await_approval | do_not_retry_blind_write

Challenges we ran into

• Component vs product narrative. Early demos looked like a mini-app. We kept the story as an agent-callable gate over existing ProDocuX / PDX Engine dependencies.
• MCP write-back schema drift. Live add_tags required tag_urns + entity_urns; older aliases failed validation. We fixed the gate’s MCP client so agent write-back actually lands.
• Honest failure modes. Fixture disclosure and missing owners yield completed_with_review even when a tag write succeeds - agents must surface review, not treat every non-crash as “done cleanly.”
• Dependency boundaries. Wire Kernel and Engine as libraries; don’t fork the whole stack into the demo agent.

Accomplishments that we're proud of

• An end-to-end agent-capable MCP path: read context → gate → approve → real add_tags, visible in the DataHub UI.
• Agents (or CLIs acting as hosts) get a stable GateResult with explicit status and host_action.
• Transport flexibility without changing the embed point: fixture / GraphQL / MCP.
• Optional deterministic evidence via existing ProDocuX, disclosed as profiles, not as fake AI comprehension.
• A fake host example that shows the correct policy: never bypass the gate with direct DataHub mutations.

What we learned

• “Agents that do real work” on DataHub means state-changing tool use with a choke point, not longer chat transcripts.
• MCP is how the agent reaches the catalog; the gate is how you keep that work accountable.
• Reusing ProDocuX and PDX Artifact Engine beats rebuilding deterministic plumbing inside the agent demo.
• Hosts need explicit status / host_action more than another dashboard.

What's next for PDX DataHub Gate

• Optional HTTP façade for hosts that should not vendor Python.
• Richer write-back (terms / owners) still behind the same approval gate.
• Deeper evidence presets while keeping ProDocuX as an external Kernel dependency.
• More multi-host templates (CI + public wheels) so agent embedders copy less bootstrap.

Built With

• cursor
• datahub
• fastapi
• gemini
• json-schema
• mcp
• openai
• pdx
• prodocux
• pydantic
• pytest
• python
• uv