UpgradePilot

Inspiration

Dependency bots open a pull request that goes red and then stop. The breaking changes live in a migration guide that the tooling never reads, so a person still has to read it and fix the code. We wanted an agent that does that whole loop and then proves it did it honestly.

What it does

UpgradePilot takes a project and a breaking version bump, finds and reads the official migration guide, edits the code, runs the project's own tests on the real target version, and loops until they pass.

The bundled demo upgrades a small order service from pydantic 1.10.27 to 2.14.2. 21 tests pass before the bump; after it, 3 collection errors block every test and more failures appear after the first fix. The agent repairs it to 21 of 21.

The result is a diff, the cited sources, a failures-per-step chart and a measured token and cost ledger (about $0.05–0.09 per run).

How we built it

1. Baseline: run the tests on the old version to prove a green start.
2. Bump: upgrade the dependency; the same tests now break.
3. Research (Tavily): search the official docs (restricted to docs.pydantic.dev) and extract the migration page. Nemotron 3.5 Lightning on Nebius Token Factory condenses it into a checklist for the failures we actually saw.
4. Repair (Nemotron 3 Super on Nebius Token Factory): a tool-calling loop (read_file, write_file, run_tests, run_python, search_docs) that edits only app/**/*.py and re-runs the tests.
5. Verify: deterministic integrity checks, so a green result is not a faked one. Tests run with --noconftest, every file outside app/ is hashed, the installed version must be the target, the pass count must equal the baseline count, and no deprecated v1 API or pydantic.v1 shim may remain.

The app is FastAPI with server-sent events and a single-page UI, in a Docker image with both environments prebuilt, deployed on Render. There are 96 unit tests (no paid calls) and GitHub Actions CI. Token counts come straight from each response's usage; prices come from Token Factory's model catalog, and dollar figures are labelled as estimates.

Results

Same task, same 40-turn limit: Nemotron 3 Super passed 5 of 5 benchmark runs (mean about $0.08 per run); Lightning passed 0 of 5 (cheaper, but it stalls after fixing part of the code); Ultra passed 0 of 3 at 6–9x the cost of a Super run. In QA on the deployed site, 5 of 5 live runs passed. Small samples on one task, reported as such.

Challenges

• Our first demo used .dict() and .parse_obj(), which v2 only deprecates, so it would have stayed green. We rebuilt the project around real breaking changes and verified the failures before writing any agent code.
• Stopping the agent from faking a green: a security review showed it could try to steer the test result through files it wrote, so we added the hash checks, the --noconftest run, import bans inside app/ and write restrictions.
• Retrieval quality: unrestricted search returned blogs and a different project's changelog, so we pinned research to the official docs domain.
• Run-to-run variance and cost tails: we put a spend cap and a turn limit on every run.

What we learned

Cheap models stall on multi-file repairs, and a bigger model is not automatically better. Input tokens dominate agent cost, and restricting retrieval to official docs matters more than the model choice for research quality.

Limitations (honest)

This is dependency isolation, not a security sandbox. It handles one bundled migration, Python only. A passing test suite is not proof that behaviour is preserved, so review the diff. A live run takes about 2.5 to 4.5 minutes on the free Render instance, which can also be slow on first load.

What's next

Arbitrary repositories in a real container sandbox, more migrations, opening pull requests, and a router that escalates between Nemotron sizes only when a cheaper model stalls.

Built With

• docker
• fastapi
• github-actions
• html
• javascript
• nebius-token-factory
• nvidia-nemotron
• pydantic
• pytest
• python
• remotion
• render
• server-sent-events
• tavily
• uv