Payne

Payne helps data teams know when a dataset is truly safe to shut down, and catches when that changes.

Comment

Payne

Approval that expires when reality changes.

Payne helps data platform and analytics engineers retire a legacy dataset without breaking the reports, pipelines, or models that still depend on it. It uses DataHub to find remaining dependencies, records the conditions required for shutdown, and withdraws the current approval when those conditions stop being true.

The hosted app uses a fixed, labelled scenario. It does not connect to a visitor's DataHub instance. The repository also contains a local workflow that runs against the bundled DataHub scenario.

The problem

Deleting a table or view takes seconds. Knowing whether its consumers have moved can take weeks.

Data platform teams often investigate a retirement across a data catalog, SQL repositories, tickets, messages, and meetings. They need to know which dashboards and pipelines still use the old dataset, whether the replacement returns the fields those consumers expect, who must approve the change, and whether the approved migration was actually applied.

The final decision usually lives in a ticket. That ticket captures one moment. A new production consumer can appear later. A schema can change. Ownership can move to another team. The old approval still looks valid even though the facts behind it have changed.

This matters when a source system must be shut down, a warehouse contract is ending, or a compliance deadline requires the old data path to close. Keeping both paths alive costs money and leaves migration work unfinished. Turning the old path off too early can break business operations.

What Payne does

Payne creates a Shutdown Gate, a record of the conditions that must remain true before a specific dataset can be retired. A gate includes the dataset, its replacement, the accountable owner, the evidence from DataHub, the required approvals, and the latest result.

The main flow is:

  1. Choose the dataset to retire and its confirmed replacement.
  2. Record the shutdown goal, deadline, owner, and required conditions.
  3. Read DataHub lineage, schemas, ownership, and available operating context.
  4. Reduce the downstream graph to the dependency that currently blocks shutdown.
  5. For the supported case, prepare a small SQL compatibility change and test each known requirement separately.
  6. Keep human review, application, and verification as separate states.
  7. Open the gate only when the recorded conditions pass.
  8. Recheck open gates in a background worker. If relevant evidence changes, preserve the prior decision and mark its approval invalid.

The language model helps interpret evidence and explain the blocker. Code-based rules decide the gate result. The model cannot approve a shutdown or turn a failed check into a pass.

The demo to remember

The core proof is the same approval before and after one fact changes.

On Day 1, DataHub evidence shows that the declared conditions pass. The gate reads Open under current evidence and monitoring is active.

On Day 2, a new production consumer appears downstream of the retiring dataset. Payne records the changed evidence, checks it against the condition that no active production consumers remain, and changes the result to Previous approval invalidated. The original approval remains in the history, but it no longer appears current.

This is the product's central idea: a shutdown decision stays valid only while the facts that justified it remain true.

Try the guided simulation

The interactive walkthrough takes about two minutes and requires no account or setup.

  1. Open the Dependency drift simulation.
  2. Use the button at the bottom of each stage to inspect the evidence boundary, the affected consumers, the decisive blocker, and the compatibility change.
  3. At Day 1: Gate Open, confirm that all three conditions pass and monitoring is active.
  4. Select Advance to Day 2.
  5. Follow the new lineage evidence through interpretation and rule evaluation.
  6. At Deterministic invalidation, confirm that the new consumer violates the recorded condition and that the previous approval is invalid.
  7. Open Notification and receipt to inspect the saved result and its limitations.

Success looks like this transition:

Day 1: Open under current evidence
Day 2: Previous approval invalidated
Reason: New production consumer detected
Required action: Resolve the dependency and record a current approval

The walkthrough is a fixed scenario with checked-in sample evidence. Use the local workflow below to verify the infrastructure-backed path.

What is working now

Hosted guided simulation

  • A labelled, fixed scenario covering the retirement objective, evidence boundary, impact review, blocker, SQL compatibility change, application receipt, open gate, changed evidence, invalidation, and notification.
  • Two additional failure cases for a bad migration output and a change in the accountable owner.
  • Downloadable sample receipts that identify their fixture and limitations.

Local workflow in the repository

  • PostgreSQL-backed gate, evidence, approval, application, monitoring, and invalidation records.
  • A bundled DataHub scenario with lineage, schema, ownership, and description evidence.
  • A bounded SQL compatibility change for the checked-in finance consumer.
  • Separate checks for SQL syntax, allowed statement shape, referenced fields, output names, types, nullability, and fixture output.
  • Application limited to the bundled sandbox, followed by DataHub re-ingestion and fresh evidence reads.
  • A worker that monitors open gates and records invalidation when the Day 2 consumer appears.
  • A reusable safe-dataset-deprecation coding-agent skill with tests.

Why DataHub matters

DataHub is the source of the facts that can open or invalidate a gate: dataset identity, downstream lineage, schema fields, ownership, and available operating context. Payne turns those changing catalog facts into a decision with explicit conditions and history.

Without DataHub, the product would have neither the dependency graph needed to find the blocker nor the later evidence change needed to withdraw approval. Payne complements the catalog rather than recreating it.

For the supported migration case, DataHub schema evidence also constrains a small SQL change. For example, a replacement field can be aliased to preserve the output name required by a known consumer. The result remains a reviewable patch until the local sandbox records application and fresh evidence confirms the intended change.

Technical boundary

Payne separates responsibilities so an AI explanation cannot become shutdown authority:

DataHub evidence
      |
      v
Agent interpretation and cited blocker
      |
      v
Code-based checks and gate result
      |
      v
Human approval and recorded application
      |
      v
Background monitoring and invalidation

The hosted simulation replays checked-in evidence. In local mode, the application stores gate state and receipts in PostgreSQL, reads the bundled DataHub instance, applies the accepted patch only inside the finance sandbox, and runs monitoring in a separate worker. Gemini and Ollama are supported for the bounded interpretation step. The same rule code owns the final gate state in either provider path.

The technically difficult part is preserving the connection between a decision and the exact evidence, approval, patch, and conditions that produced it. Payne versions those records so a later graph or ownership change can invalidate the current approval without erasing its history.

Limits

  • Payne can only evaluate dependencies represented in the connected evidence sources. Missing metadata remains an explicit gap.
  • An open gate supports a human decision. It does not authorize or delete production data.
  • The migration workflow supports one target dataset, one confirmed replacement, and the documented SQL compatibility case.
  • Fixture checks prove the declared examples and output contract. They do not prove every production input or the business meaning of the data.
  • Accepting a patch does not mean it was applied. Payne requires a separate application record and fresh evidence.
  • The hosted app demonstrates product behavior with fixed data. Live DataHub reads, writes, persistence, and worker monitoring require the local workflow.

Run the local proof

For the shortest no-infrastructure path, clone the repository and run:

pnpm install --frozen-lockfile
pnpm dev

Open http://localhost:3000/simulation?case=dependency-drift&step=objective.

The full local workflow requires Node.js 22 or 24 LTS, pnpm 11, Docker Compose v2, 2 CPUs, 8 GB RAM, 13 GB free disk, and the fixed ports listed in the local quickstart. Follow that guide for credentials and startup. Once the stack is ready, run:

pnpm db:seed
pnpm local:verify
pnpm local:day2

The seed command prints a Shutdown Gate URL. pnpm local:verify checks the Day 1 scenario. pnpm local:day2 adds the production consumer for the invalidation path. The expected final result is a persisted history entry showing that the new dependency invalidated the previous approval.

To verify the public release, including links, types, tests, samples, documentation, and the production simulation build, run:

pnpm release:static

Why this fits the challenge

  • DataHub use: DataHub evidence determines the blocker and provides the changed facts that can invalidate an approval.
  • Agent work: The agent investigates a bounded retirement, cites the relevant evidence, explains the blocker, and prepares the supported compatibility change.
  • Metadata-aware code generation: The generated SQL is constrained by recorded schemas and one known consumer contract. Its validation limits stay visible.
  • Technical execution: The result depends on versioned evidence, explicit authority boundaries, independent monitoring, and a reproducible Day 1 / Day 2 transition.
  • Open source: The Apache 2.0 repository includes the application, local scenario, tests, sample receipts, and reusable coding-agent skill.

The takeaway

Payne turns a one-time dataset retirement approval into a monitored decision that expires when its evidence changes.

Built With

Share this project:

Updates

Submission history