Understudy

Inspiration

You can pin a library forever; you can't pin a model past its retirement date.

On 2026-09-30 Anthropic deprecated Claude Sonnet 4.5 (claude-sonnet-4-5-20250929) and set its retirement for 2026-11-30, with claude-sonnet-5-5 as the replacement. Three weeks later, API version 2026-10-21 removed temperature, top_p and top_k, and the Python SDK 2.0.0 started sending that version by default. On 2026-10-23, OpenAI shut down 17 legacy model entries listed on its deprecations page, among them gpt-4, gpt-4-turbo, o1 and o3-mini.

We wrote a small accounts-payable service the way many teams wrote structured extraction in 2025: Anthropic SDK 0.71.0, temperature=0, a forced tool call and content[0].input. Then we tried the obvious fix and changed the model string. Claude Sonnet 5.5 returned HTTP 400 tool_choice: type "tool" and "any" are not supported for this model., and Claude Haiku 5.5 returned HTTP 400 `temperature` is deprecated for this model. Upgrading only the SDK fails too: 1.x no longer accepts temperature, and 2.x sends the API version that rejects it. A model swap is a behaviour change, not a string change.

OWASP's Top 10 for LLM Applications 2025 lists "Outdated or Deprecated Models" under LLM03 Supply Chain. Most teams still handle a model retirement as a calendar reminder and a one-line diff.

What it does

Understudy keeps an AI feature working through its model's retirement, and the whole process runs in GitLab:

1. A scheduled pipeline reads the provider's official retirement table and compares it with ai/models.lock.yml.
2. It opens an issue, due two weeks before the retirement date, and a draft MR.
3. It starts a GitLab Duo flow that works through Anthropic's migration checklist in the MR: SDK pin, no temperature, structured outputs in place of the forced tool call, content blocks read by type, refusals handled.
4. CI auditions every candidate model on extraction quality, cost per 1,000 invoices, latency and refusals. Each case runs several times because sampling can no longer be pinned.
5. CI runs payment-redirection and prompt-injection attacks through the full payment decision. Each attack class that succeeds becomes a vulnerability in GitLab's security widget: CWE-345 with OWASP LLM06 Excessive Agency for redirection, and CWE-1427 with LLM01 or LLM07 for injection and prompt leakage.
6. When a gate fails, the same flow is restarted in repair mode with the failures as its goal.
7. When every gate passes and a cheaper candidate passed them too, the flow switches the feature to it once, and CI auditions the switched configuration again.
8. When everything passes, Duo Code Review runs. After merge, a person approves once, and CI publishes a GitLab Release with a Model Change Record.
9. An hourly probe sends test invoices and an attack through the released build to the live model, and opens an incident when it fails.

The patient app, Payables Desk, reads supplier invoices with Claude and decides whether a payment can be scheduled or must be held for a clerk. The audition found a real hole. The legacy rule schedules a payment whenever the model says no review is needed, and it never compares the IBAN with vendor master data. We sent Claude Sonnet 4.5 a spoofed invoice from a known vendor that simply printed a fraudster's IBAN, with no injected instruction. The model extracted the account, mistyping one digit, and marked the invoice as fine, so the payment would have gone out. Understudy reports this as a security vulnerability, and the agent's repair moves the payee check into code.

How we built it

• GitLab Duo Agent Platform: one custom flow (AI Catalog item) with two modes, migrate and repair, guided by two skills and AGENTS.md. Flows can't be triggered by a pipeline, so CI starts them through the Flows API with a human steward token. Duo Code Review is requested the same way.
• GitLab CI/CD: the spine. It runs tests, a lifecycle policy gate, a guard that stops agents from editing protected paths, the audition and red team per candidate, GitLab SAST and secret detection, a decision step, packaging, the manual approval, the release and the scheduled probe. Results go into JUnit, metrics and SAST reports, so they show in the MR's own widgets.
• Google Cloud (written, not run): the pipeline has a Cloud Run path (staging and production in europe-north1 with a 10% canary, Artifact Registry, Cloud Build, Secret Manager, and Workload Identity Federation so CI deploys without keys) that takes over once its variables exist. Our Google Cloud project had no billing account before the deadline, so that path hasn't run and we don't claim the Google Cloud bonus.
• Python and FastAPI for the app and the python -m understudy toolkit, with offline tests that use a fake Anthropic client.

Challenges we ran into

• Only humans can fire flow triggers. A pipeline can't mention or assign the flow's service account to start a session, so CI calls the Flows API with a human token. We look up the AI Catalog consumer ID with GraphQL.
• Flow jobs get no CI/CD variables. The agent can't hold the API key, and it shouldn't. Every real model call happens in an ordinary CI job, and the agent works against offline tests.
• Sampling can't be pinned any more. With temperature gone, the audition runs each case several times and judges it by its worst sample.
• Responses can start with thinking blocks. Code that reads content[0] breaks without raising an error, so the migrated code reads blocks by type.
• The default Duo image has no pip. The first agent session stopped in setup with No module named pip. The flow now runs in python:3.12, set in .gitlab/duo/agent-config.yml, which GitLab reads only from the default branch.
• No Google Cloud billing in time. The Cloud Run path waits for its variables, so we added a path that needs no cloud: a manual approve:release job, a release that carries the Model Change Record and the prompt pack, and an hourly probe that starts the released build inside the job and checks it against the live model.
• The participant role has limits. It can't create protected environments, enforced CODEOWNERS approval, schedule variables or alert integrations. We used a manual approval job and advisory CODEOWNERS, and we document the gap.

Accomplishments that we're proud of

• The third full run went from the retirement notice to the merge in 15 minutes with no human action: the agent migrated the feature, the red team caught two payment-redirection attacks on Claude Sonnet 5.5 (a Critical CWE-345 finding), the agent repaired the code, and the policy picked Claude Haiku 5.5, which passed every gate at $0.36 per 1,000 invoices against $4.79 for Sonnet 5.5.
• A real migration MR written by a Duo flow, for a real retirement date.
• A payment-redirection finding that appears in the security widget like any other SAST finding, and is resolved by the agent's repair.
• One person makes one decision, and the decision is recorded in the release.

What we learned

• Most of the work in a model migration is checking behaviour, not changing code.
• Attacks on an AI feature fit GitLab's vulnerability workflow well: severity, a stable identity across pipelines, and resolution after a fix.
• Duo Code Review earned its place. It noticed that our policy switch renamed the MR to the new model but left the old one in the description, so decide now updates both.
• Splitting the work by trust boundary worked well. The agent writes code, CI holds secrets and measures, and a person approves.

What's next

• Watchers for other providers' retirement tables and for partner platforms whose dates differ from the provider's own.
• Protected environments and required Code Owner approval on projects where the role allows them.
• Several features in one lock, so related migrations can be grouped into one MR.
• Publishing the migration skill and flow in the AI Catalog for other projects.
• Running the Cloud Run path (staging, canary, rollback to the last revision that passed) on a billed Google Cloud project.

Built With

• anthropic-claude
• fastapi
• gitlab
• gitlab-ci/cd
• gitlab-duo-agent-platform
• gitlab-package-registry
• python