ModelLedger

"Dependabot for AI models: scan any repo for model dependencies, check them against a lifecycle ledger, and block retired models in CI — with exact file, line, and a suggested replacement.

Comment

Inspiration

AI providers rename, deprecate, and retire models on their own schedule — but production repositories hardcode model IDs as plain strings scattered across code and config. Nothing warns you. Teams find out when a deployment fails or behavior silently changes. Package dependencies solved this problem years ago with Dependabot; model dependencies have nothing. ModelLedger treats AI models as what they are: dependencies that need inventory, lifecycle checks, and CI enforcement.

What it does

ModelLedger scans a repository, finds every AI model reference with exact file and line, resolves each against a lifecycle registry (active / deprecated / retired / unknown), assigns deterministic risk, and returns CI-friendly exit codes — 1 blocks the merge on a critical finding, 2 signals input errors, 0 passes. It suggests the recommended replacement for deprecated and retired models. Output is human-readable or JSON, via CLI or a Streamlit dashboard. The full loop: retired model detected → merge blocked → swap in the suggested replacement → rescan → pipeline green.

How we built it

Built solo during Build Week using Codex in VS Code as the implementation agent, spec-first: every milestone was a written prompt with explicit scope and a "do not commit" rule — every commit was human-reviewed and human-made. The pipeline is deliberately deterministic and auditable: AST-based parsing for Python, conservative structured parsing for JS/TS/JSON/YAML/TOML/env templates, a YAML registry with lifecycle metadata, and a rule-based risk engine. Same input → same findings → same exit code.

A dedicated Codex /review pass surfaced 11 defects (3 high, 5 medium, 3 low). A remediation cycle plus an independently executed verification battery — wheel install into a clean venv, comment false-positive tests, symlink containment, malformed-registry and exit-code contract checks — gated the first commit. The suite grew from 15 to 49 passing tests.

Challenges

Scanner precision: The first version used global string matching — a retired model mentioned in a comment created a critical finding and failed CI. False positives destroy trust in CI tooling, so we rebuilt detection around key-anchored contexts: comments, docstrings, and prose strings can never produce findings.

Packaging: Editable installs worked while the built wheel silently omitted the registry — caught only by installing the wheel into a clean venv and running from outside the source tree. That test is now permanent.

Containment: Symlinks could escape the scanned repository, including a discovery-to-read race — fixed with containment re-verified at read time.

What we learned

Precision beats recall for CI tools; a review-gated agent workflow catches what a fast build misses; and "boring and deterministic" is a feature you can demo.

What's next

A verified registry of real provider records — every lifecycle fact carrying a source URL and verification date — then modelledger.lock, policy-as-code, a GitHub Action with SARIF/PR annotations, and a grounded GPT-5.6 migration planner that interprets verified evidence and cites official sources.

Built With

Share this project:

Updates