Inspiration
Deleting a database column is easy. Knowing whether you are actually safe to delete it is the hard part.
A field can look completely dead in Git while still powering a dashboard, a scheduled query, a Spark job, a notebook, or an ML feature owned by another team. Most tools can show pieces of that blast radius. They do not finish the job: migrate the consumers, prove the replacements work, check again for anything new, and only then permit the destructive change.
That gap is where Retirement Conductor began.
We wanted to answer one practical question:
Can we retire this legacy field right now—and can we prove why?
Not with a confidence score. Not with a stale approval. With fresh evidence and a real, enforceable decision.
What it does
Retirement Conductor coordinates the complete retirement of a legacy data field.
An operator declares that an old field has been replaced. Retirement Conductor then:
- Uses DataHub to discover downstream consumers across systems.
- Records the scope, freshness, pagination status, permissions, and blind spots of that evidence.
- Produces an exact migration plan for authorized Git and dbt consumers.
- Pauses for human authorization before making changes.
- Opens a reviewable change and validates it with native tools such as dbt and CI.
- Rechecks DataHub from scratch instead of trusting the original inventory.
- Publishes the campaign result back into DataHub and reads it back.
- Issues short-lived, one-use authority for the producer-side retirement.
- Revalidates the evidence immediately before executing the real PostgreSQL change.
- Executes the retirement—or refuses it if the world has changed.
In our representative campaign, repository search found one consumer. DataHub found 31 consumers across seven pages of lineage. That was not additional decoration for a report; it changed what the system was allowed to do.
The most important moment happens after the migration appears complete. We introduce a new Spark consumer that still reads the legacy field. Fresh DataHub lineage discovers it, the campaign changes from READY_TO_RETIRE to UNSAFE, the existing Retirement Lease is invalidated, and the PostgreSQL column is preserved.
We also ran the unsafe control. With static sign-off, the column was dropped and the Spark workload failed with PostgreSQL error 42703: the legacy column no longer existed.
That contrast is the heart of Retirement Conductor: the answer is allowed to change when the evidence changes.
How we built it
DataHub is the cross-system context and reconciliation layer.
Retirement Conductor uses its graph to inspect field-level lineage, schemas, ownership, usage context, and downstream relationships that cannot be discovered reliably from a single repository. It also publishes a durable campaign summary back into DataHub so the final decision becomes part of the shared operational context rather than disappearing into a terminal session.
The core campaign engine is deliberately deterministic. It owns authorization checks, evidence freshness, state transitions, validation acceptance, lease issuance, replay protection, and the final retirement decision.
Around that engine, we built:
- A project-scoped MCP server exposing the Retirement Conductor workflow.
- A Codex skill that teaches an agent how to operate a retirement campaign safely.
- Git and dbt migration planning with exact-file authorization.
- Native dbt parsing, builds, tests, semantic validation, and CI binding.
- Fresh DataHub reconciliation and publication/read-back.
- A Superset integration proof with native chart execution and compensation.
- A separately privileged PostgreSQL producer action that performs a real
DROP COLUMN. - One-use, digest-bound Retirement Leases that expire or become invalid when evidence changes.
- Durable recovery for interruptions and unknown mutation outcomes.
- A Workbench backed by the same campaign state used by the CLI and MCP interfaces.
Codex acts as the operator interface and orchestrator: it investigates context, calls tools, proposes the migration, explains the evidence, and carries the workflow forward. It cannot authorize itself or override a refusal. Consequential safety decisions remain in deterministic code, with human authorization outside the agent.
Challenges we ran into
The hardest problem was distinguishing “nothing was returned” from “nothing depends on this.”
Metadata can be incomplete because of pagination, permissions, stale ingestion, unsupported lineage, connector failure, or identity mismatches. Retirement Conductor therefore treats absence as a claim that must be qualified—not as proof of safety.
That principle exposed a real pagination problem in DataHub’s MCP server: later lineage pages could be silently hidden. We fixed the behavior at the GraphQL boundary and opened DataHub MCP PR #195. We also opened PR #196 to correct misleading deployment-gate diagnostics.
Other difficult parts included:
- Binding authorization to the exact proposed change instead of a vague approval.
- Preventing stale, replayed, expired, or tampered plans from executing.
- Validating migrations in their native systems rather than trusting API success.
- Recovering safely when a destructive request succeeds but its response is lost.
- Detecting a consumer that appears after approval but before retirement.
- Giving an agent useful freedom without giving it authority to declare its own work safe.
- Keeping an increasingly sophisticated implementation understandable as one simple product.
We also had to resist adding AI where it did not help. A controlled ablation across 135 attempts showed that a nested Gemini planning layer did not improve exact migration-plan accuracy. We removed it. DataHub context stayed; redundant model complexity did not.
Accomplishments that we're proud of
We are proud that this became more than a polished impact-analysis demo.
Retirement Conductor completed a real end-to-end campaign that:
- Inspected live-local DataHub context.
- Planned and applied an exact dbt migration.
- Stopped at an external human authorization boundary.
- Passed native dbt validation and public CI.
- Published and independently reread its result through DataHub.
- Issued one-use retirement authority.
- Executed a real PostgreSQL column removal.
- Refused replay after the authority was consumed.
- Detected a late Spark consumer in the unsafe scenario.
- Invalidated previously issued authority.
- Performed zero destructive producer actions after that discovery.
- Demonstrated the real downstream failure that static approval would have caused.
We subjected the system to frozen synthetic truth sets, adversarial cases, drift, expiration, outages, tampering, replay, interrupted operations, permission loss, and lost responses. The goal was not to manufacture a flawless demo. It was to find the conditions under which the product should refuse—and make those refusals reliable.
We are also proud that the project contributed upstream instead of only consuming DataHub. The two DataHub MCP pull requests came directly from problems encountered while building and validating Retirement Conductor.
What we learned
The biggest lesson was that refusal is not a failure mode. For this product, a correct refusal can be the most valuable result it produces.
We also learned that the agent should not be the safety boundary. Models are useful for understanding context, navigating tools, proposing work, and explaining decisions. They should not be trusted to authorize themselves, accept their own validation, or override missing evidence.
Another experiment gave us an uncomfortable but useful result: a carefully implemented fresh-check CI workflow matched Retirement Conductor on the raw safety properties we tested.
That forced us to sharpen the product instead of hiding the result.
Retirement Conductor is not claiming that ordinary CI cannot be made safe. Its value is that it joins the entire operation into one durable workflow: DataHub discovery, exact consumer migration, native validation, human authorization, fresh reconciliation, revocable one-use authority, producer execution, recovery, and a causal audit trail.
Finally, we learned that more intelligence is not always more useful. DataHub’s fresh context materially improved fault detection. Adding another LLM inside the workflow did not. The best architecture was the simpler one: an agent for orchestration, deterministic code for consequential decisions, and DataHub as the living context that can change the answer.
What's next for Retirement Conductor
The next step is not to add integrations for the sake of a longer feature list.
We want to test Retirement Conductor with independent data platform operators, measure where the workflow creates friction, and learn whether teams trust and understand its evidence. We also want to exercise it against larger, messier organizational graphs with real authentication boundaries, incomplete metadata, and production-scale ownership.
From there, we plan to:
- Strengthen production authorization and least-privilege deployment boundaries.
- Add native execution adapters only where real operator demand exists.
- Expand evidence-quality diagnostics for incomplete or stale metadata.
- Improve independent recovery across distributed systems.
- Continue contributing fixes and operational improvements upstream to DataHub.
- Turn the Workbench into a focused control surface for inspecting, authorizing, and reconciling real campaigns.
Our long-term goal is simple:
Make deleting a legacy field the final, boring step of a process that has already proved it is safe.
Built With
- ci/cd
- datahub
- datahubmcpserver
- dbt
- docker
- fastapi
- githubactions
- graphql
- mcp
- next.js
- postgresql
- python
- react
- sqlite
- typescript
Log in or sign up for Devpost to join the conversation.