Bumpless

Inspiration

Dependabot is great at noticing that pydantic 2 is out, and useless at what comes next. A major bump opens a red PR, and then someone spends an afternoon reading a migration guide, searching for old APIs, rewriting them and hoping the tests cover it. Those PRs pile up.

I wanted an agent that does that afternoon's work and shows enough evidence that a reviewer can trust it.

Dependabot bumps the version. Bumpless makes it pass.

What it does

Point Bumpless at a repo and a target version, or let the GitHub Action pick up a red Dependabot PR.

• Research. Tavily finds the official migration guide and changelog. Nemotron 3 Nano turns each page into candidate rules, and Super builds a rulebook. Every rule is checked against the page it came from, including whether the old usage really breaks. Only verified rules are applied.
• Characterize. Before touching anything, it records how the code behaves on the old version and saves that as tests.
• Migrate. Ultra plans and Nano edits files in parallel. A failed edit escalates up the tiers: Nano → Super → Ultra. Guardrails reject any edit that weakens a test.
• Verify. Tests run in a Docker sandbox. Ultra diagnoses and repairs failures, looking up unfamiliar errors with Tavily.
• Report. Verified change coverage shows which changed lines the tests actually executed:

$$ \text{verified change coverage} = \frac{\lvert \text{changed lines executed by tests} \rvert}{\lvert \text{changed lines} \rvert} $$

Every change gets a confidence rating plus the rule and source URL behind it.

• Talk. Reviewers can reply on the PR:

@bumpless revise
@bumpless explain
@bumpless revert

Comments from outside contributors are refused before any model call.

• Learn. Verified rulebooks are saved as recipes for the next repo. Radar shows a repo's upgrade backlog with risk and cost estimates.

How I built it

The pipeline:

intake → baseline → research → characterize → plan → edit → verify → coverage → review → learn

• Backend: Python, FastAPI, SQLite and a Typer CLI.
• Models: Every call goes through Nebius Token Factory (OpenAI-compatible API) to NVIDIA Nemotron 3 Nano, Super and Ultra. Each task is routed to the cheapest tier that does it well, and every call's cost is tracked and capped per run.
• Tavily: Search with raw markdown, map and extract, all cached in SQLite. Tavily powers source verification, recipe freshness checks and error lookup.
• Frontend: React, Vite and Tailwind, with a live run view that shows every step, model call, tier, cost and Tavily query as it happens.
• Sandbox: Docker, non-root, with CPU and memory limits and network access only for installs.
• GitHub Action: A composite action for Dependabot PRs, plus a comment job for @bumpless replies.
• Benchmarks: A harness that stores every run with its commit, and replays of real runs for the hosted demo.

Challenges I ran into

• Rules that are true but optional. On SQLAlchemy, "use where() instead of filter_by()" was backed by the docs, but it only applies to the new query style, and applying it broke working code. Verification now also asks whether the old usage really breaks.
• "Reply NONE if nothing applies." This was a strong attractor for Nemotron with reasoning off. Nano and Super said NONE in 8 of 9 tries on a guide page full of before-and-after code. A reworded prompt got 5 of 5.
• Reasoning budgets. With reasoning on, Nano and Super sometimes spent their entire 16k-token budget thinking about a single edit. Turning reasoning off for edits cut the median run time from 286 s to 134 s, a reduction of \( 1 - \tfrac{134}{286} \approx 53\% \).
• Running out of credit mid-hackathon. When the Token Factory credit ran out, I moved to the NVIDIA API catalog free tier. That meant dealing with models that are listed but not callable on a trial key, and with rate limits.
• Honest benchmarks. Results vary between runs of the same code, so I publish every run and its commit, not just the best one.

Accomplishments that I'm proud of

• pydantic v1 → v2 and openai 0.28 → 1.x migrations passed on all 5 test repos: 37 of 37 broken tests fixed, with every change cited to an official source and covered by tests.
• Tiered routing beat "Ultra for everything" on both cost and quality: \$0.398 vs \$0.564 on the NVIDIA free tier, and 5 of 5 repos passing vs 3 of 5.

$$ \text{savings} = 1 - \frac{0.398}{0.564} \approx 29\% $$

• A real Dependabot PR fixed end to end by the GitHub Action. It detected the red major bump, migrated the code on the Actions runner, pushed the commit and posted the report. Tests went from 0 → 32 passing with 93.5% verified change coverage, for about \$0.075.
• SQLAlchemy 1.4 → 2.0 with no saved recipe passed: 6 of 6 repo tests and 26 of 26 recorded behaviors, with 100% verified change coverage. I also published the runs that failed on the way there.
• Recipe reuse. The second repo with the same pydantic migration took 161 s instead of 394 s.

What I learned

• Grounding beats memory. Nano answering from memory invented pydantic changes. Fetched and verified sources fixed that.
• A cheap model is only cheap if its first attempt works. Most of my spend went to Ultra repairing failed edits, so better first edits are the biggest lever for cutting cost.
• Small prompt wordings can flip results completely. Example IDs in prompts also get copied by the model, so placeholders matter.

What's next

• Benchmark on real-world repos with pinned commits. The harness already supports them.
• Deploy on Nebius Serverless and add a Serverless Jobs sandbox executor.

Built With

• dependabot
• docker
• fastapi
• github-actions
• httpx
• nebius-token-factory
• nvidia-nemotron
• nvidia-nim
• openai-sdk
• pydantic
• pytest
• python
• react
• recharts
• render
• server-sent-events
• sqlite
• tailwindcss
• tavily
• typer
• typescript
• uv
• vite