Upkeep

Upkeep works your whole pull-request queue: it plans the merge order, fixes what a policy allows with NVIDIA Nemotron on Nebius Token Factory, and proves every fix with your compiler and tests.

Comment

Try it: live demo (press "Work the queue"; it calls Nemotron on Nebius Token Factory live) · code (MIT) · where it can be wrong

Inspiration

Every team has the same quiet backlog: pull requests that are one small change away from merging. The branch is behind. A conflict touches a single line. A rename on main broke the build. A Dependabot bump no longer compiles. Nobody is blocked for a good reason, and nobody has the next twenty minutes.

Bots made the pile bigger, not smaller. A study of Dependabot on GitHub found developers receptive to its pull requests, but 11.3% of projects had already dropped it, and the authors' list of what an ideal dependency bot needs reads like a spec: configurability, autonomy, transparency and self-adaptability (arXiv:2206.07230). I have triaged bot pull requests for a living, and the pattern is always the same: the fix is small, the context switch is not.

An agent can make these changes. The hard part is trusting it: how far may it go, who checks its work, and what did it cost? Upkeep is built around those three questions.

What it does

Upkeep works through a repository's open pull requests as one queue:

  1. Plans the queue before touching code: which pull requests change the same files, which pairs would conflict (git merge-tree), and a suggested merge order. After the run it checks the ready ones again and flags any the run's own merges made stale.
  2. Detects what blocks each one by merging the base branch into a scratch copy and running build, vet and tests in a container with no network.
  3. Forecasts (dependency upgrades) what will break: Tavily finds the migration guide at runtime, Super reads it with the release notes, and later marks every prediction confirmed or not seen by the build.
  4. Fixes what a bounded policy allows: mechanical merges and formatting without a model, small conflicts with Nemotron Ultra in one call, broken builds and tests with Nemotron Super and four tools, up to three attempts, each from a clean tree.
  5. Verifies outside the model: build, vet and tests pass; at most 5 files and 80 lines; no test touched; Go files only; no comment added; formatted. A conflict in a test or a non-Go file is never resolved, and comments in Ultra's resolutions count too. A change to a module in a subdirectory, which the root checks cannot build, goes to a person.

  6. Acts within the policy: push the fix to a bot's branch or one labelled upkeep, otherwise attach a patch; merge a bot's patch or minor bump when everything passes; never merge a person's pull request; never force-push; hand back what it cannot do with Ultra's diagnosis.

  7. Learns. Every fix Super made that passed the verifier is remembered with its failure; when the same failure returns, Upkeep reapplies it with zero model calls and still runs every check. In the demo, Run again fixes the same three pull requests for $0.0055 instead of $0.0191.

Every ready pull request gets a merge confidence (high, medium or low) with its reasons, and every pull request ends with a receipt: the models it used, calls, tokens, time and cost, and for an upgrade the forecast with its marks and sources, next to the same tokens priced at the largest model (a price comparison, labelled as one; it does not claim the largest model would have fixed it).

The hosted demo runs a repository with nine pull requests built to show one blocker each (behind main, a one-line conflict, an import conflict, a rename that breaks the build, a major, a minor and a patch bump, formatting with an order-dependent test, and a large conflict). One press of "Work the queue" and four are worked at a time, with a live board, a models rail with the cost of the run, and a side sheet per pull request with the forecast, the diff, the verifier and the receipt.

The same engine works real repositories: upkeep run -repo owner/name lists the open pull requests, recognizes bots, reads a Dependabot bump from its title, pushes fixes, posts receipts, and arms GitHub's auto-merge so GitHub waits for the checks. On our sample repository Dependabot opened three real pull requests, and all three merged. One, a clockwork 0.4 to 0.5 bump whose release note says FakeClock became a struct, broke the build: Upkeep had Super fix it in one attempt for $0.0017, pushed the fix with its receipt and armed auto-merge.

The dashboard shows the agent at work: a live trace per pull request (the plan, the Tavily search, every Super turn with the tool it called and the file it edited, each attempt and why it failed, the verifier, the action), a merge-order graph, and a models rail with the cost of the run.

Upkeep also ships as a GitHub Action (uses: aalvaropc/[email protected]) that runs on a schedule, from the Actions tab or from a /upkeep comment, and keeps one updated receipt comment per pull request. Installed on our sample repository, it brought the other two up to date, saw that they conflict with each other in go.mod, let GitHub merge one, then had Ultra resolve the other's conflict in one call for $0.0006; both are merged, each with one receipt (details).

Why it matters, and what is new. About one merge in five conflicts, and conflicted code is twice as likely to carry a bug (Brindescu et al., EMSE 2019); unmerged Dependabot security pull requests are mostly blocked by concurrent changes (Alfadel et al., MSR 2021). Today's tools fix one pull request at a time when asked, predict risk without fixing, or order pull requests that are already green. Upkeep treats the backlog as one job: it plans it, forecasts before building, fixes within a policy, proves the fix outside the model, learns from it, and prices every step.

How we built it

One Go binary is the command line, the server and the dashboard (React and TypeScript, embedded). The standard library does nearly everything.

The routing is measured, not assumed. Before writing it we ran 73 experiments (3.06 million tokens, about two dollars) on Token Factory. Ultra inside a tool loop fixed 0 of 21 (it explores instead of editing and invents packages) but resolved small conflicts in one call 3 of 3 and writes the best diagnoses, so Ultra does exactly that. Super fixes builds, with reasoning off, in up to three fresh attempts within a 300,000-token budget: about 65% per attempt in our measurements. A rename on main was fixed 12 of 12. For the forecast we first used Nano; on the recorded prompts its answers varied between runs, while Super gave the same answer three times and always named the identifier the build then reported, so Super reads the forecast. Nano writes one sentence for the reviewer and that sentence is checked against the diff: if it names anything that is not in the change, it is dropped.

The model proposes; code disposes. The edit tool refuses test files and added comments; the verifier recomputes the rules on the final diff; pushes are fast-forward only and the code has no way to force-push; the sandbox has no network, no capabilities and a read-only root.

Reproducible. Everything that leaves the process goes through record and replay. The nine-pull-request run was recorded once with the real models and is embedded in the binary: with no key, replay gives the same nine outcomes and the same cost to the tenth of a cent, and a CI job replays it against the real sample repository and fails if any pull request ends differently.

Challenges we ran into

  • Nearly right is the dangerous kind of wrong. One model answered a conflict with the whole enclosing block, so lines appeared twice; another put a raw tab inside a JSON string; another counted two resolutions for one conflict. We fixed these in code (trim echoed context, repair control characters, ask once more with the reason) and kept the model's job narrow.
  • A forecast that sounds right. The first forecasts listed removals that did not exist. Grounding every prediction in the sources, giving the model the identifiers the project actually uses, switching the reader to Super, and discarding search results about other projects (one search had returned a different project's release page) fixed it, and the dashboard shows each prediction as confirmed or not seen by the build.
  • Flaky or order-dependent? The scenario's test fails in the full run and passes alone because of shared state. Retrying it only hides that, so Upkeep runs the full suite twice more: failing both times makes it order-dependent, otherwise it is flaky. Either way it is reported and left alone, and the docs say the heuristic can be wrong.
  • Making the demo safe to be public. Isolated sessions with their own copy of the repository, rate limits that ignore forwarded addresses unless a proxy is trusted, a daily token budget that falls back to the recording, and a limit on concurrent runs.
  • A tool that never answered. Diagnosing a failed live run on the last day, we found that go_doc returns nothing inside the no-network sandbox: go doc loads the whole module graph and fails on a module the download step did not fetch. Super had been fixing builds from the code, the build output and the forecast alone. Fixing it changes what the tool returns, so it would invalidate the recorded run; we documented it instead of re-recording on the last day.

Accomplishments that we're proud of

  • The whole scenario finishes in about half a minute (33 s live on the demo machine) for about two cents (eight live runs against the real models: 72 of 72 pull requests ended as expected, about two to three cents a run on average), and every pull request ends with a receipt.
  • The recording is reproducible to the tenth of a cent, and CI proves it on every pull request.
  • It works on real Dependabot pull requests, not only on a demo.
  • 507 Go tests and 83 dashboard tests, a no-comments rule enforced by the build, more than 70 pull requests merged through protected main, each with CI and the replayed evaluation, and documentation that says where the product can be wrong.

What we learned

  • Measure before you route. Our first design had Ultra at the centre of the fix loop; it was the worst model for that job and the best for two others.
  • A cheaper model is only cheaper if you can check what it says. Nano's note and Super's forecast are both checked against something that is not a model.
  • The comparison people expect ("all Ultra") is a price, not a result. We print it as a price.
  • Bounded autonomy is a feature users can read: the rules fit on one panel.

What's next

Always-on webhooks so Upkeep reacts when a bot opens a pull request; a GitHub App with scoped permissions; other languages behind the same verifier; the sandbox on Nebius Serverless Jobs; learned recipes for recurring dependency breaks; and a public results page for the live evaluation.

Built With

  • caddy
  • docker
  • github-actions
  • github-api
  • go
  • nebius-token-factory
  • nemotron-3-nano
  • nemotron-3-super
  • nemotron-3-ultra
  • nvidia-nemotron
  • react
  • tavily
  • typescript
  • vite
Share this project:

Updates

Submission history