Bailiff - Agent Court for DataHub

The courthouse for AI agents on DataHub: register identity, gate MCP writes, cross-examine catalog vs SQL, propose changes, write verdicts to GMS so the next agent inherits.

Comment

Inspiration

Agents can read DataHub now. They still cold-write metadata, trust catalog descriptions over the warehouse, and leave no verdict on the graph when SQL disagrees.

We built Bailiff so agents do governed work: register identity, gate mutations, cross-examine claims against SQL, turn risky writes into proposals humans accept, and write verdicts back so the next agent inherits truth not chat memory.

What it does

Bailiff is a courthouse for AI agents on DataHub.

Register · Gate · Examine · Propose · Inherit

  • Register - Each agent is a first-class identity (local docket + best-effort aiAgent emit to GMS).
  • Gate - bailiff mcp sits in front of MCP-shaped tools; unregistered or over-scope mutations are DENIED.
  • Examine - The Claim Examiner runs SQL on DuckDB; when catalog text lies (e.g. “gross” vs warehouse net), the verdict is CONTRADICTED and can write back to the graph.
  • Propose - Governed stewards cannot silently rewrite the catalog; changes become proposals; a human accepts, then apply runs.
  • Inherit - Ask the same business question twice: without Bailiff the analyst trusts bad metadata; with Bailiff the answer uses the court verdict (net revenue).

Context Moat - Five fixed scenarios (S1–S5), score 1.0, replayable with bailiff demo --replay (no Docker).

Live DataHub - When GMS is online, examine and accept paths write tags and structured properties; API responses include writeback_mode: datahub. bailiff demo --live fails closed if GMS is down.

In the demo UI: Run governed agent (deny → examine → propose → accept → WITH/WITHOUT compare) and Run moat suite.

How we built it

Stack

  • Python bailiff_core - court, policy, DuckDB examiner, SHA-256 hearing ledger, moat harness, GMS write helpers
  • FastAPI bailiff_proxy - hearings, proposals, health/writeback, POST /agent/run for one-click governed loop
  • bailiff mcp - stdio MCP gate before search, lineage, and mutations
  • Next.js Court UI - Operations board, governed agent run, moat runner, analyst compare panel
  • agents/mcp_steward.py - deterministic steward through the gate
  • Fiction retail world in DuckDB for reproducible examine/inherit demos

Architecture

LLM / MCP client → Bailiff MCP gate → Court (policy + examiner + ledger)
                        │                    ├── DuckDB SQL probes
                        ▼                    └── GMS write-back (when online)
                 DataHub MCP / GraphQL → DataHub GMS

Quickstart for judges (~2 minutes, no DataHub required)

git clone https://github.com/AmanM006/Bailiff
cd Bailiff
python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
pip install -e ".[dev]"
bailiff demo --replay
bailiff serve
cd apps/court-ui && npm i && npm run dev
pytest -q

Open http://localhost:3007 - Run governed agent / Run moat suite. Frozen proof: examples/moat_report.json.

Optional live path: DataHub Quickstart, set DATAHUB_GMS_URL and DATAHUB_GMS_TOKEN, then bailiff demo --live.

Source: https://github.com/AmanM006/Bailiff

Challenges we ran into

  1. OSS vs Cloud - Quickstart does not expose every Cloud propose_* GraphQL path. We shipped an OSS-compatible proposal docket (pending → accept → apply) that still updates the local mirror and GMS when reachable.
  2. Credibility - Early UI booted from fixtures and looked like a slideshow. We removed fake defaults; the board stays empty until the real API is up.
  3. Inherit proof - The analyst WITH/WITHOUT panel had to flip only after accept and verdict write-back, not from querying the wrong table or wrong scoring weights.
  4. Seeding - Re-seeding on every UI action wiped accepted catalog state. We split soft seed from user-driven actions so demos do not undo their own story.

Accomplishments that we're proud of

  • End-to-end deny → examine → propose → accept → inherit with an honest WITH/WITHOUT analyst compare on a fixed revenue scenario
  • Fails closed live demo when GMS is offline
  • Context Moat 1.0 replayable without Docker
  • Real MCP gate (bailiff mcp) plus Court API and UI judges can click through in one session
  • Upstream contributions to datahub-skills:
    • PR #93 — fix dangling /datahub-audit routing hint in datahub-enrich
    • PR #96 — new datahub-bailiff skill (register / gate / examine / propose / inherit)

What we learned

DataHub is a strong context platform, but production agents need identity, policy gates, and graph-durable verdicts not just read access. Governance belongs in the agent stack (MCP + skills), not only in a separate dashboard.

What's next for Bailiff - Agent Court for DataHub

  • Merge datahub-bailiff skill upstream
  • Optional hosted judge demo (no local clone)
  • Deeper Cloud proposal / mutation APIs where GMS exposes them
  • More moat scenarios and enterprise policy packs

Built With

Share this project:

Updates