Arbiter

Inspiration

DataHub already gives AI agents context — lineage, ownership, quality, glossary. What it doesn’t do is own the decision. Schema and dbt PRs still ship on gut feel and Slack threads; agents still query sensitive tables because they can. After an incident, the “why we blocked this” rarely becomes durable metadata, so the same failure repeats. We wanted a system that doesn’t just answer questions about the graph — it judges change and agent action, then writes that judgment back so the organization accumulates institutional memory, not tribal lore.

What it does

Arbiter is an autonomous change & agent control plane for DataHub. It sits on the critical path of data changes (PRs, schema/dbt diffs) and AI agent tool calls, then runs a closed loop:

Signal → Read DataHub → Decide (ALLOW / DENY / ESCALATE) → Act → Write-back → Verify

Scores blast radius from real lineage, ownership, tags, and policy
Blocks unsafe merges or quarantines rogue agent actions
Writes every verdict and rationale back to DataHub so the next human or agent inherits why, not just what
Runs in enforce, shadow, or dry_run modes with a production console, GitHub webhook checks, and an agent gateway SDK

How we built it

FastAPI arbitration service (/v1/arbitrate/change, /v1/arbitrate/agent) with JWT/API-key auth
DataHub client package for live GMS reads/writes, plus offline fixtures for reliable demos
Versioned YAML PolicyPack for hard rules — the LLM explains and proposes remediation; it never owns the gate alone
Next.js console for verdict timelines, demo launcher, agents, policies, and API keys
GitHub webhook + Check Runs for PR arbitration; arbiter_gateway SDK to gate agent tools pre-call
Optional Anthropic/OpenAI enrichment for grounded rationales; Prometheus metrics and structured logs for ops

Challenges we ran into

Making demos work with and without a full DataHub quickstart — fixtures vs live GMS without lying about the product
Mapping messy PR/asset names to stable DataHub URNs so lineage and owners actually resolve
Keeping risk scoring transparent and policy-driven instead of a black-box LLM “feels risky”
Designing write-back so every run leaves the graph different — verdicts as first-class metadata, not chat logs
Balancing autonomy with trust: high blast radius must escalate to humans, not auto-merge everything

Accomplishments that we're proud of

A working end-to-end loop: breaking PR and rogue-agent demos that produce real allow/deny/escalate verdicts
Deep DataHub use on both sides — read for evidence, write for institutional memory
Agents as first-class subjects, not just pipelines and dashboards
A shippable console + API + CLI + gateway SDK, not a slideware agent
Policy-first design that judges can audit: hits, risk factors, and rationale grounded in URNs

What we learned

Context without authority still fails. Impact analysis is evidence; enterprises need a verdict. Write-back is the product bar — if the graph looks the same after a run, you built a chatbot. Hard policy + grounded LLM explanation beats either alone. And treating AI agents like production consumers (registry, purpose, attestations) is how “context for agents” becomes safe self-serve in practice.

What's next for Arbiter

Richer DataHub aspects for ChangeVerdict / attestations and prior-verdict memory that compounds risk over time
Deeper CI integrations and post-deploy verify loops tied to freshness, contracts, and incidents
Stronger MCP / Agent Context Kit / Skills packaging so any agent can gate through Arbiter by default
Broader change types (Airflow/Dagster DAGs, warehouse schema events) and ML lineage gates for model→deploy risk
Shadow-mode rollouts in real data platforms so teams earn trust before flipping to enforce