Repo Surgeon

Point Repo Surgeon at any GitHub repo. It upgrades dependencies, patches CVEs, and proves its own fixes work with mutation testing, then opens small, risk graded pull requests, unattended.

Comment

Inspiration

Every repo we've worked on has thirty outdated dependencies, a few unpatched CVEs, and a test suite nobody trusts enough to run before merging a fix. We wanted to see if an agent could own that whole loop, not just suggest a diff, so we built Repo Surgeon on Codex.

What it does

Point Repo Surgeon at a GitHub repo and a five-stage pipeline runs unattended inside an isolated Docker sandbox:

  1. Scout clones the repo, detects the stack, and records a pass/fail baseline plus a security scan (OSV-Scanner, pip-audit, npm audit).
  2. Researcher uses GPT-5.6 with web search to pull real changelogs and migration guides for each outdated or vulnerable dependency.
  3. Surgeon dispatches upgrades to Codex running headless, editing code and re-running affected tests after each change.
  4. Verifier re-runs the full suite against baseline, loops failures back to the Surgeon, then runs mutation testing on any new tests to prove they'd actually catch a regression.
  5. Reviewer splits the work into small, risk-ordered PRs with evidence and a confidence grade, then watches CI and pushes fix commits on failure.

How we built it

  • Orchestrator: Python/FastAPI state machine, plus GPT-5.6 for planning, risk ordering, and PR writeups.
  • Agent runtime: Codex CLI headless (codex exec), with subagents for parallel upgrades and a post-edit hook that re-runs affected tests.
  • Sandbox: one Docker container per job, network locked to allow-listed registries.
  • Verification: full suite re-run plus mutmut (Python) and Stryker (JS/TS) for mutation testing.
  • GitHub layer: GitPython + GitHub REST API for branches, risk-graded PRs, and CI polling.
  • Dashboard: Next.js + Tailwind with a live SSE feed.

One owner per layer, nightly 20-minute syncs.

Challenges we ran into

  • Token blowup on research calls. Input tokens dominated output roughly 20:1 with web search on. Capping output tokens broke generation, since reasoning tokens share that budget. Fixed by batching 3 packages per call plus a pacing gate.
  • Four people, one sequential pipeline. Solved by defining shared Pydantic contracts and Protocol interfaces up front, so real implementations swapped in for mocks without touching the orchestrator or dashboard.
  • One-shot Codex edits were unreliable. Built a bounded retry loop (edit, re-test, diff against baseline, retry with failure context) capped at 5 attempts, then flagged needs_human instead of force-merging.
  • Generated tests can pass trivially. Added mutation testing scored against mutation score, coverage, and stability, so green means the tests would actually catch a regression.
  • Sandboxing untrusted code. Docker with resource/capability/mount limits plus phase-based network policy (network during install, none during execution).
  • Python 3.9 compatibility gap surfaced mid-integration and was fixed during the Researcher/Reviewer/CI-watcher pass.
  • Multi-job dashboard state. Fixed cross-job SSE leakage and a stale-job-ID crash after backend restart, both found through manual end-to-end testing.
  • Real mode is opt-in. Mock mode is default everywhere; live GitHub/OpenAI calls need explicit opt-in and credentials. CI repair is capped at 2 fix commits per PR.

At submission: 58 passing backend tests, all four production stages enabled in real mode, dashboard verified end-to-end, and a live Codex smoke test on a real bump (requests==2.31.0 to 2.32.3).

What we learned

  • Token cost is an input problem once web search is involved, not an output one. Profile where tokens actually go before optimizing.
  • Never trust a single LLM edit. A verify-and-retry loop with a hard cap and an honest needs_human state beats one-shot generate-and-merge.
  • Passing tests isn't evidence of correctness. Mutation testing is the cheapest way to check if generated tests would catch a real regression.
  • Contracts before implementations was the biggest unlock for four people building in parallel against a sequential pipeline.
  • Default to the safe mode and make risk explicit. Sandboxing untrusted code has to be designed alongside the pipeline, not bolted on.
  • Cap everything that could loop, or a stubborn failure becomes an unbounded cost sink instead of a clean signal.
  • Manual end-to-end testing surfaces bugs unit tests don't; some things only show up when you use the product like a user would.

What's next

Scoped from what's explicitly still open at submission time, not aspirational ideas.

Immediate: pick and authorize 2 to 3 demo-fork repos (enabled but not yet exercised), build the Docker sandbox images (docker/python/Dockerfile, docker/node/Dockerfile exist but weren't built as of 2026-07-20), and get the real demo fork URL into the video walkthrough.

Near-term hardening: wider language/stack detection beyond Python and JS/TS, hostname-level network policy (currently phase-based), broader mutation testing coverage, and guaranteeing scanner tools are present rather than silently degrading.

Product direction: multi-repo batch mode, a review UI for needs_human items, per-job cost/token visibility on the dashboard, and graduating Planner.from_openai() to the default (behind the same real-mode gate).

If this became a real product: persistent job storage (currently in-memory), auth/multi-tenant support, and rate-limit-aware scheduling extended across the whole pipeline.

Built With

Share this project:

Updates