Nightshift

Inspiration

Version-bump bots are a solved problem. Dependabot and Renovate find the vulnerable pin and open the pull request, and they do it well.

Then they walk away. A large share of those pull requests fail CI, because the patched version moved an API. Someone has to read the traceback, work out that Session.request() no longer takes timeout positionally, rewrite the call site and push. That human step is why security upgrades sit open for months at organisational scale — not the bump, the repair.

We wanted to know whether an agent fleet could do the part that is left over, and do it in a way a security team could actually let run overnight without watching it.

What it does

At 02:00 Cloud Scheduler wakes a scanner. One batched OSV.dev query covers every pinned dependency across the whole fleet. Each affected repository becomes one Pub/Sub message, and Cloud Run workers fan out — one container per repository, up to twenty at a time.

Each worker runs the same five phases:

Baseline — run the repository's own suite untouched. Nothing passes → the repository arrived broken. Fewer than half pass → our environment is wrong, and the verdict says so rather than blaming the repository. Above that it is usable, and the tests that were already red are recorded by name and set aside.

Upgrade — rewrite the manifest to the fixed version.

Verify — run the suite again. The break is what the upgrade changed, not what was red when we arrived.

Repair — if something broke, a bounded Gemini 3.5 Flash loop reads the traceback and rewrites the call site. Every tool call passes through a policy engine first.

Pull request — opened on a fork, authorship disclosed, nothing merged.

Then the Librarian — an agent with no tools at all, by construction — reads the finished repair and generalises it into a migration recipe scoped by {library, from_version, to_version}. The fortieth repository to hit jinja2 2.11 → 3.1 starts from the answer instead of the traceback. Cost per repository falls as the fleet works.

A human opens the control tower at 08:00 and reads what happened.

How we built it

Google Cloud end to end: Cloud Run Jobs for the scanner and the per-repository workers, Pub/Sub for fan-out, Cloud Scheduler for the 02:00 trigger, Firestore for job state and the Ledger of record, Secret Manager for credentials, Cloud Build for images, Cloud Trace + OpenTelemetry for the cost curve, and a Cloud Run service for the dashboard.

The agents run on ADK 2.0 against Gemini 3.5 Flash on Vertex AI. Four agent roles are designed; two are built and running — Repair and Librarian. Triage is a deterministic severity floor today rather than a Gemma pass, and the Reviewer agent is specified but not built. The README says so in the same table that lists them, because a submission that quietly implies otherwise is the easiest thing in the world to write and the hardest to defend.

Three properties carry the trust story, and each is a code path with tests holding it there rather than a sentence in a prompt:

The agent cannot touch the tests. An agent asked to turn a red suite green will, given the chance, delete the failing test. Writes to test files and to CI configuration are denied by the policy engine — the most heavily tested module in the repository, a pure function of (ToolCall, Budget) with no I/O and no clock. Baseline first, always. Without a run of the suite as it arrived, every number we report would be taking credit for breakage we did not cause. Failure is a first-class result. Outcome is a closed enum in which UNBUILDABLE and BASELINE_RED are ordinary members. Repositories we could not help stay in the denominator, which is what turns "repair rate" from a claim into a number.

Forks by default, ALLOW_UPSTREAM_PRS=false, no auto-merge implementation and no flag that enables one, every loop with a ceiling on attempts, wall-clock and tokens.

Challenges we ran into

Almost every bug lived in the seam between our code and somebody else's system. The domain logic — the phase machine, the policy engine, the Ledger — never broke. Four that cost the most:

Every pull request the fleet earned died on one line. Fifty-one repositories in a single night were cloned, built, measured, upgraded and tested, and then failed at git commit with "Author identity unknown": a container has no git identity. The test suite was green throughout, because the fixture wrote a user.email into its own clone — it was testing a repository that had the one thing a fresh clone does not. The fix is four words on a command line. Finding it took a fleet run and a log.

The model was never reachable from Cloud Run at all. ADK's synchronous runner drives its generator on a second event loop; paired with an asyncio.run for the session, a long-lived worker ended up minting credentials through an executor that had already closed. ADK swallows that and yields events anyway, so it arrived as "the model produced no answer and no token usage" — an infrastructure outage filed as though the agent had been asked and had nothing to say.

A retry policy with no ceiling is not resilience. Two repositories returned pytest exit 3 on every single delivery. We had classified that as our fault and nacked the message, so they came back for ever: thirty-one of fifty-two finished jobs in one night were the same two repositories, each costing a full container and a full environment build, while forty others waited behind them.

The candidate search excluded the population the project exists for. Both queries asked for pushed:>2025-06-01. A repository touched in the last three months keeps its dependencies current; a current dependency has no advisory whose only fix is a major version away. We had built a fleet to repair breaking upgrades and then carefully selected repositories that could not produce one.

Accomplishments that we're proud of

The fleet runs, unattended, in the cloud, against repositories nobody curated for it: 144 verdicts across the deployed runs and 14 pull requests open on real repositories. The most recent full pass over the 39-repository pool reached the upgrade on 11 of them.

The repair loop is proven end to end on a known break: jinja2 2.11.3 → 3.1.2, where 3.0 removed the top-level Markup re-export and the suite fails during collection. Repaired on the first attempt, 8,238 tokens, one line changed, tests untouched, pull request open.

And we are proud of the numbers that are not flattering, because we kept them. The first version of our results table read 0 reached the upgrade; 51 of 57 repositories never got as far as being asked. We published that, then spent the rest of the project closing it.

What we learned

Reading does not find these bugs. Running does. Every failure above was invisible to a green test suite and obvious within one minute of a real fleet run.

The same rule implemented twice drifts apart. The worker and the fleet probe once disagreed about what a red baseline meant, and while they disagreed the fleet threw away thirty-five usable repositories and filed our container's limitations as theirs. Firestore and the in-memory store disagreed about where a run id lived, so every filtered query came back empty and the dashboard could only show every night at once. Each of those is now one function with one test.

The bottleneck in automated dependency repair is not the repair. It is reproducing somebody else's environment well enough to run their tests. And in this population, buildability and breakage turn out to be anti-correlated: the repositories that reproduce cleanly are modern and safe to upgrade, and the ones carrying a dangerous pin are the ones whose environment no longer exists.

The expensive part is the cheapest part. Total model spend across the whole project is a rounding error next to the containers. A repair costs about eight thousand tokens on Flash. The architecture is built so the model is only reached after everything cheap has already narrowed the problem, and the measurements bear that out.

What's next for Nightshift

Zero wild repairs so far, and we know exactly why. OSV answers with the lowest version carrying a fix, which is usually a patch release, and patch releases break almost nothing — all fourteen pull requests are upgrades that left the suite green. The fleet is for the minority where the only published fix is a major version away. We have just added a search band for repositories nobody has touched in one to three years, which is where those advisories live; the next step is widening it until the intersection of buildable and breaking is large enough to measure a repair rate over.

Then: read the build recipe out of the repository's own CI instead of guessing at pip install; detect repositories that need a database or a browser and say so honestly rather than spending a container to call them broken; move the Ledger onto Vertex AI Memory Bank, which is one implementation of one protocol away; and build the two agents that are specified and not yet written — Gemma triage and the Reviewer.