Upgradient

Explore breaking dependency upgrades in parallel Nebius sandboxes; ship one tested migration with evidence.

Comment

Inspiration

Upgrading dependencies is easy to start and hard to finish. A major version bump breaks the app, and someone has to work out the migration and prove it works. Bots open version-bump pull requests and stop there. Their PRs are accepted about half as often as human ones, and 80% of application dependencies go un-upgraded for over a year although newer versions exist. We wanted an agent that finishes the job and shows its work.

What it does

Give Upgradient a pinned Python project and upgrade targets. It:

  1. Proves a green test baseline in Nebius Token Factory Sandboxes (full suite, run twice).
  2. Runs two or three upgrade strategies in parallel sandbox forks: a minimal fix, an API migration, a co-upgrade. For a new Python version it rebuilds the runtime first.
  3. Repairs what breaks. Nemotron 3 Super triages failures and writes patches that cite migration guides found by Tavily; Nemotron 3 Ultra reviews every edit before it is tested and rejects fabricated output.
  4. Rebuilds the image from the exported lock and patch only, re-runs the full suite, checks that the tests are unchanged and re-scans advisories.
  5. Exports the best strategy as a git branch and a pull request in which every claim names the sandbox operation or source behind it, plus an exploration tree of every path tried.

Results:

  • httpbin, 2018 lock, Python 3.6 → 3.12 with Flask/Werkzeug 3.1: 67/67 tests, all 20 target advisories resolved (including a HIGH Werkzeug debugger advisory), 24 application lines changed, $0.36 of inference.
  • records, Python 3.6 → 3.12 with SQLAlchemy 2.0: 39/39 tests after five chained breakages, no test changed.
  • httpbin, Flask 1.0 → 2.0: 4 of 5 runs passed, one of them live on the judge site.

How we built it

  • Token Factory Sandboxes run every build, install and test. Strategies fork the same saved image in parallel. Operation IDs are recorded before waiting, so a crashed run re-attaches instead of starting over.
  • NVIDIA Nemotron on Token Factory: Super for triage, patches and test adaptations; Ultra for confirming "the test tool changed" claims and reviewing every edit. We measured the smaller models and kept them for the setup check only: Nano 1/14 and Lightning 0/9 vs Super 5/6 on triage. On review, Ultra scored 9/9 on labelled patches vs Super 6/9.
  • Tavily finds migration guidance during the run. Results are cached and cited with fetch times; the project's own repository is excluded, so the model isn't handed the maintainers' later fix.
  • Python for the orchestrator, OSV for advisories, and a small FastAPI judge site that serves runs, free replays and capped live runs.

Challenges we ran into

  • Models cut corners. Super proposed editing assertions and injecting the header a test expected. So the guardrails live in code: patches cannot touch test files, and Ultra reviews every edit.
  • A broken app vs a changed test tool. Werkzeug's new test client stopped sending Content-Length: 0, which two tests check. A test may change only through a small catalogue (such as adding content_length=0), with assertions identical in the syntax tree and the adapted test still passing on the old environment. Ultra still mistook application bugs (bytes shown as text) for test-tool changes, so a deterministic check now overrules it.
  • Runtime migrations. On Python 3.12 the suite doesn't even import. Upgradient measures progress by the import error, probes the installed package for where removed names went, and upgrades locked packages that no longer build.
  • Chained breakages. In records, each SQLAlchemy 2.0 fix revealed the next error, and stale tracebacks misled the model until Upgradient re-triaged on changed errors.
  • Still hard: the Python 3.12 target passed 1 of 2 runs on the final code, and in one failed run Ultra accepted an edit that no test covers. Failed runs are reported as failed.

Accomplishments that we're proud of

  • A real 2018 project moved to Python 3.12 and current Flask, verified on an image rebuilt from exactly the exported files.
  • Every claim in the PR traces to a sandbox operation or a fetched source, and every path tried is kept.
  • Parallel strategies measurably help: on the same inputs and budgets, three in parallel passed 4 of 5 runs and one strategy passed 2 of 5.
  • Judges can replay a recorded run for free or start a capped live run.

What we learned

  • Guardrails belong in code, not prompts.
  • Targeted context beats whole files: Super found the right test-client fix only from library excerpts selected by the failing value.
  • Reviewers need the baseline. Once told that a test passed before the upgrade, Ultra went from rejecting a correct fix 4 of 4 times to accepting it 4 of 4.
  • Model output varies from run to run; parallel strategies turn that variance into a higher pass rate.

What's next for Upgradient

  • More projects, and live runs on pinned requirements.txt files (supported, tested offline so far).
  • Later upgrade waves that continue from the previous winner, and bisecting across version bumps.
  • Opening the PR on an authorised fork, and a remediation report per advisory.

Built With

Share this project:

Updates

Submission history