InheritBench

Move the model. Keep the capability. InheritBench helps developers transfer fine-tuned behavior to replacement models and prove whether the migration is ready to ship.

Comment

Inspiration

Companies are gaining more freedom to switch between open-weight models for cost, latency, licensing, privacy, and infrastructure reasons. But the behavior they fine-tuned into one model does not automatically move to the next.

A replacement model can still sound intelligent while losing the hard-earned capability created through fine-tuning: structured decisions, policy handling, approval logic, tool selection, and safety behavior. Generic benchmarks do not answer the operational question that matters:

Did the capability my system depends on actually survive the migration?

Satya Nadella’s idea of the “reverse information paradox” helped sharpen the motivation. As AI becomes more abundant, the examples, corrections, policies, and operating knowledge inside an enterprise learning loop become more valuable. I wanted to build a tool that helps developers retain ownership of that learned capability when the underlying model changes.

That became InheritBench.

What it does

InheritBench is a local CLI for model succession.

A developer defines the capability their current model must preserve using a capability pack containing:

  • model-visible examples;
  • evaluator-only expected contracts;
  • input and output schemas;
  • controlled vocabularies;
  • safety rules;
  • coverage groups;
  • readiness thresholds;
  • direct labels and optional intervention anchors.

The developer then points InheritBench at an adapted source model and a replacement target model.

InheritBench:

  1. validates the capability pack;
  2. verifies that the source model actually has the capability;
  3. measures what the untouched target lost;
  4. runs a configured recovery strategy;
  5. trains successor candidates;
  6. selects a candidate using validation evidence only;
  7. opens sealed clean and adversarial tests after selection;
  8. applies a deterministic readiness contract;
  9. exports the recovered adapter;
  10. verifies it on a fresh base-model load;
  11. produces replayable evidence and a migration decision.

The final outcome can be:

  • PASS
  • CONDITIONAL_PASS
  • MIGRATION_BLOCKED

InheritBench does not assume that recovery succeeds. Blocking an unsafe or underperforming migration is a valid result.

The reference succession

I used OpsRoute as the first real capability pack. OpsRoute covers structured refund-policy routing and subscription cancellation or retention decisions, including approval requirements and unauthorized-action checks.

The source was an adapted Qwen2.5-0.5B-Instruct model. The replacement was OLMo-2-0425-1B-Instruct.

Because Qwen and OLMo are different model families with incompatible adapter shapes, the original Qwen LoRA adapter could not simply be attached to OLMo.

The untouched OLMo target failed the capability diagnostic with 0 / 32 semantic successes.

A direct recovery strategy improved the model but undercovered an important scenario group. InheritBench identified the supervision deficit and requested ten targeted original anchors. I then ran four bounded seeded candidates using the same supervision, optimizer, schedule, validation surface, and readiness contract.

Candidate 0 was selected using validation evidence only. The clean and adversarial final surfaces remained sealed until after selection.

The selected successor achieved:

  • clean operational correctness: 64 / 64;
  • clean exact-contract fidelity: 63 / 64;
  • clean strict validity: 64 / 64;
  • clean safety blockers: 0;
  • adversarial exact-contract fidelity: 20 / 32;
  • adversarial strict validity: 31 / 32;
  • two safety findings on one adversarial record;
  • final readiness: CONDITIONAL_PASS.

The adapter passed fresh-base reload verification, and the final decision was replayed across 192 saved predictions.

How I built it

I built InheritBench as three connected layers.

Capability-pack layer

The capability pack separates model-visible inputs from evaluator-only answers. It also defines schemas, safety predicates, scenario coverage, allowed labels, anchors, and readiness thresholds.

This allows the succession engine to operate on a declared capability contract instead of hardcoding OpsRoute logic into the framework.

Succession CLI

The CLI supports capability creation, validation, inspection, succession planning, execution, intervention, resume, inspection, replay, and browser export.

The main workflow is:

uv run inheritbench capability validate capabilities/opsroute/v0.2.0

uv run inheritbench succession plan \
  --pack capabilities/opsroute/v0.2.0 \
  --source-config configs/models/source.yaml \
  --target-config configs/models/target.yaml \
  --strategy anchored-behavioral-transfer-v0.1 \
  --output runs

uv run inheritbench succession run \
  --plan runs/<run-id> \
  --device mps

Planning freezes a content-addressed execution plan before model compute begins. Execution persists stage-level evidence, prevents final-test leakage, selects checkpoints using validation only, and fails closed when inputs or artifacts do not match the frozen plan.

Browser evidence surfaces

The completed-run inspector visualizes the real Qwen-to-OLMo succession produced by the CLI.

The Assurance Lab is a lighter browser-accessible testing surface. It does not train models or run fresh Qwen or OLMo inference. Instead, it evaluates generated predictions, checks schemas and safety rules, applies readiness logic, verifies integrity, supports controlled mutations, and replays the result.

For example, changing one prediction to bypass approval moves the decision from:

CONDITIONAL_PASS → MIGRATION_BLOCKED

using the same evaluator and readiness contract.

Challenges I faced

The hardest challenge was separating a successful experiment from a trustworthy succession process.

It would have been easy to train until the final numbers looked good. Instead, I had to enforce validation-only selection, sealed final evaluation, immutable plans, no-overwrite evidence, and explicit data-role separation.

I also encountered a numerical-guard failure during the four-seed recovery run. The original guard treated large but finite pre-clipping gradient norms as instability, even though clipping had succeeded. I repaired the guard to distinguish pre- and post-clipping behavior and to fail only on genuinely non-finite training state. The repair did not change the supervision, seeds, optimizer, schedule, checkpoints, selection rules, readiness rules, or final surfaces.

Another challenge was product framing. The browser Assurance Lab initially made the project look like a frozen-results dashboard. I restructured the product and documentation so the hierarchy is explicit:

  • the CLI is the developer product;
  • capability packs define what must survive;
  • the reference run proves the engine executed;
  • the Assurance Lab lets judges test the assurance layer without rerunning accelerator-backed training.

What I learned

I learned that model migration is not simply a model-quality comparison.

A successor can preserve operational decisions while differing on exact policy codes. It can pass clean scenarios while still failing adversarial safety checks. It can also produce fluent, schema-valid output while losing the actual capability.

That required me to keep several concepts separate:

  • operational correctness;
  • exact-contract fidelity;
  • strict structural validity;
  • safety findings;
  • scenario coverage;
  • readiness.

I also learned that reproducibility depends on details that are easy to overlook, including adapter initialization, random-number-generator state, deterministic schedules, candidate-selection boundaries, and exactly-once final evaluation.

Most importantly, I learned that trustworthy AI tooling should be able to say no. InheritBench is useful not only when it recovers a capability, but also when it produces MIGRATION_BLOCKED and preserves the evidence explaining why.

Accomplishments I am proud of

I am proud that InheritBench is not only a dashboard or an LLM wrapper.

It includes:

  • a working local CLI;
  • developer-authored capability packs;
  • real Qwen and OLMo model execution;
  • LoRA recovery training;
  • bounded multi-seed candidate generation;
  • validation-only selection;
  • sealed clean and adversarial evaluation;
  • deterministic readiness decisions;
  • adapter export and fresh-load verification;
  • model-free replay;
  • browser-side evaluator parity;
  • integrity checks;
  • controlled mutation testing;
  • extensive unit, browser, accessibility, documentation, and mobile tests.

Codex and GPT-5.6 helped me implement and audit the engine, experimental controls, evidence pipeline, Python-to-TypeScript evaluator parity, browser product, tests, and documentation. Deterministic code owns every metric, candidate selection, safety finding, and readiness gate.

What is next

The current real execution boundary is intentionally narrow: the pinned Qwen2.5-0.5B to OLMo-2-1B registry, the structured-JSON capability profile, and Apple MPS as the demonstrated training backend.

My next steps would be:

  1. execute a second real capability pack;
  2. add another validated target-model architecture;
  3. prove live generic teacher generation;
  4. test additional accelerator backends;
  5. expand the clean and adversarial evaluation surfaces;
  6. package capability-pack authoring into a smoother enterprise developer workflow.

The long-term goal is simple:

Move the model. Keep the capability. Prove it survived.

Built With

Share this project:

Updates