LedgerLens — Policy-Sealed Incident Commander

DataHub grounds the plan; deterministic policy authorizes the exact work; receipts preserve the handoff.

Comment

Inspiration

A data alert says something looks wrong. It does not reliably answer which DataHub asset is involved, who owns it, what is downstream in lineage, which facts are known versus unknown, which actions are safe and reversible, who authorized the exact payload, or where the receipts and open questions live for the next responder.

A general chatbot can make that gap worse: fluent model output is useful for proposing and criticizing work, but it must not silently become execution authority. LedgerLens makes evidence, uncertainty, authority, actions, and handoff visible in one incident workspace.

What it does

LedgerLens runs one bounded incident-command turn:

trigger → DataHub context → bounded plan → AI advisory verification → deterministic authorization → receipted actions → DataHub write-back → next-agent handoff

  • DataHub context: retrieves owner, tier, schema, documentation, quality signals, and bounded downstream lineage. Facts keep source references; unknowns stay explicit.
  • Bounded plan: proposes only allowlisted, reversible collaboration work. It does not claim rollback or recovery.
  • AI advisory verification: structured verifier variants inspect evidence coverage, unknowns, scope, targets, parameters, and risk. AI may object; it may not authorize.
  • Plan-exact authorization: ordinary Python checks the exact reviewed plan fingerprint, action schema, evidence IDs, verifier preconditions, risk, and target allowlists. A changed tool call, target, or stale grant closes the gate.
  • Receipted actions: typed GitHub, Slack, PagerDuty, and Jira adapters use preview/execute separation, signed authorization binding, idempotency, bounded retries, conservative timeout handling, and sanitized receipts.
  • DataHub write-back: a disabled-by-default mutation path records the bounded incident receipt through an allowlisted DataHub MCP tool.
  • Next-agent handoff: preserves known facts, unknowns, completed work, receipt references, and the checks required before anyone can claim recovery.

Three judge-testable artifacts

1. Seal Lab — change the plan, lose the grant

The public Space includes an interactive, server-evaluated Seal Lab. Start with the reviewed plan, then append an unreviewed tool call, introduce a verifier objection, or change a Slack destination outside the allowlist. The browser calls the real deterministic gate code and displays the reviewed and evaluated fingerprints, failed conditions, reason codes, and a copyable redacted JSON receipt.

The lab is deliberately a labeled fixture: provider tools remain held, externalMutations is false, candidateOnly is true, and canClaimAGI is false. It demonstrates authorization behavior, not production safety or incident recovery.

2. DataHub Context Cut — remove facts, watch authority disappear

Context Cut makes DataHub necessity falsifiable rather than decorative. It holds one recorded model plan and its exact seal fixed, then lets a judge remove synthetic ownership or lineage facts. The current server re-runs the deterministic PolicyGate using per-tool evidence contracts: the full map authorizes, while removing primary-owner or blast-radius facts withdraws authority and marks affected tools held.

The planner and verifiers are not re-run, no provider tool executes, and no external mutation occurs. This demonstrates that required DataHub facts are load-bearing for authorization of that fixed plan; it does not claim adaptive re-planning, validated uplift, production reliability, or incident recovery.

3. Live Evidence Ladder — no collapsing separate runs into “production”

The public page now places three evidence classes side by side:

  • E-16: one supervised invocation planned, verified, authorized, and executed one bounded action against GitHub, Slack, PagerDuty, and Jira.
  • E-07: a separate controlled DataHub OSS document write and official-MCP read-back.
  • E-21: repeated credential-free samples of the deployed fixture, Seal Lab refusal, and Context Cut decisions.

The ladder SHA-256-binds the judge-facing receipt views to the archived source receipts, verifies the shared incident identity, and explicitly reports integratedSameProcessRun: false. E-21 executes no provider tool and is not an uptime SLO or provider-reliability result. The still-missing combined DataHub read → provider act → write-back receipt is displayed as an open gap rather than hand-waved away.

How we built it

DataHub is load-bearing on both sides. The read path grounds ownership, schema, documentation, quality evidence, and lineage-derived response scope through the official DataHub MCP surface. The write path records a bounded receipt so the next agent can retrieve the same entity and provenance instead of reconstructing state from chat history.

Authority remains outside the model. Planner and verifier roles use typed JSON contracts and an allowlisted tool catalog. Each tool can require specific DataHub fact IDs; the planner sees those requirements, and the deterministic gate enforces the same contract. A model can draft, abstain, or reject a plan, but authorization is issued only by deterministic policy over the frozen plan. Approval is not a loose human-in-the-loop “yes”; it is a grant bound to a specific fingerprint, scope, evidence contract, and claim boundary.

The public Hugging Face Space contains no provider credentials or live model keys. Every simulated action receipt is labeled fixture://. CI covers Python 3.11/3.12, Ruff, strict mypy, deterministic tests, secret scanning, public-package checks, readiness checks, and a non-root container build. A separate no-secret continuity workflow takes five time-separated public samples by default and uploads only a sanitized aggregate receipt.

Challenges

  • Binding authorization to the exact plan and payload while keeping model output advisory.
  • Making DataHub facts determine tool eligibility without turning metadata presence into a truth claim.
  • Keeping provenance distinct from truth or validation.
  • Handling ambiguous provider timeouts without blind retries.
  • Keeping fixture replay, recorded-model output, deterministic replay, local-live evidence, and bounded provider work visibly separate.
  • Making an agent do useful work without weakening the non-model authority boundary.

Accomplishments

  • A typed trigger-to-handoff incident commander with DataHub context and write-back structurally necessary rather than decorative.
  • Three judge-visible proofs: Seal Lab makes plan-exact refusal falsifiable; Context Cut makes DataHub fact requirements falsifiable; Live Evidence Ladder makes the evidence ceiling falsifiable.
  • Tool-specific DataHub evidence contracts shared by the agent tool catalog and deterministic policy gate.
  • Four typed provider adapters with authorization binding, idempotency, and sanitized receipts.
  • One bounded four-provider rehearsal, a separate DataHub write-back plus MCP retrieval receipt, and repeated no-provider-tool public contract samples.
  • Deterministic context-ablation and real-policy-gate evidence with explicit limitations.
  • An upstream DataHub MCP provenance/audit-context issue #159 and PR #160; the PR remains open and no upstream acceptance is claimed.

What we learned

The most valuable role for AI in an incident commander is often proposing and criticizing a bounded plan, not owning authority. Once DataHub context, evidence IDs, action schemas, and plan fingerprints are explicit, deterministic policy can enforce a much narrower execution contract.

“Real work” also needs receipts and next-step semantics. A provider response proves an action result, not root cause or recovery. A DataHub write-back is useful when it carries unresolved questions and tells the next agent what evidence must arrive before the status can change.

What is next

Complete consented formative reviews without inventing results; record one integrated DataHub read → sealed provider action → write-back → MCP read-back sequence when an owner-controlled instance is available; respond to upstream review on PR #160 without bypassing maintainers; and verify recovery only from new DataHub observations rather than treating an action receipt as resolution.

Limitations and claim boundary

LedgerLens is a working prototype. It does not establish provider-family independence, adaptive re-planning in Context Cut, sustained or production provider operation, incident causality, user impact, recovery, production reliability, security certification, independent validation, validated uplift, upstream acceptance, or AGI.

candidateOnly: true
canClaimAGI: false

Built With

Share this project:

Updates