Suture

Suture closes the wound in failing code: Nemotron on Nebius Token Factory finds the bug, tests parallel patches in a sandbox and keeps the smallest stitch that holds. Your tests stay untouched.

Comment

Inspiration

A failing test is the most precise bug report a codebase will ever give you, yet turning it into a fix is still manual: read the traceback, find the file, patch, rerun, repeat. LLMs can write patches, but one guessed patch is often wrong. Worse, an agent left alone can "fix" a red suite by editing the tests. I wanted an agent that closes the loop: it runs its own patches, learns from failures, and is structurally unable to cheat.

What it does

Suture takes a Python repository with failing pytest tests and returns a minimal patch, a plain-language explanation, and a full event log.

  1. Runs the suite in a locked-down sandbox and parses every failure.
  2. Localizes the fault with spectrum-based fault localization. A tiny coverage plugin records which lines each test executes, and each line is scored with the Ochiai coefficient:

$$\text{Ochiai}(s) = \frac{e_f(s)}{\sqrt{F \cdot \left(e_f(s) + e_p(s)\right)}}$$

Here \(e_f\) and \(e_p\) count the failing and passing tests that run line \(s\), and \(F\) is the number of failing tests. On the demo repo it ranked the exact buggy line first.

  1. Nemotron Nano condenses raw pytest output into a compact JSON triage.
  2. Nemotron Ultra writes a diagnosis and a unified diff: three candidates in parallel on Token Factory.
  3. Every diff is validated before it touches anything. It must stay inside the repo, leave test and config files alone, contain no test-gaming code, change at most 60 lines, compile, and pass git apply --check. Rejections go back to Ultra as precise errors.
  4. Nano reviews each patch for special-casing. Candidates that are the same program merge into votes, and the distinct ones are tested in parallel sandboxes.
  5. The best patch is applied and the suite reruns. Regressions are reverted, the history feeds the next attempt, and a winning patch is minimized to what the tests need.

A web UI streams every step live, with the model, tokens and latency per call. A replay mode plays back recorded real runs.

How I built it

  • Models: NVIDIA Nemotron-3-Ultra-550b-a55b (hard reasoning: diagnosis and patches) and NVIDIA-Nemotron-3-Nano-30B-A3B (frequent, cheap calls: triage and patch review). Both are served by Nebius Token Factory through its OpenAI-compatible API with the stock openai SDK. The client honors Retry-After on 429s and uses Token Factory's JSON mode for Nano.
  • Sandbox: a fresh Docker container per test run, with no network, a read-only root, memory/CPU/PID limits, all capabilities dropped, a non-root user, and a host-enforced timeout. The repo goes in as a tar on stdin, so there are no bind mounts.
  • Agent: about 4,500 lines of plain Python (FastAPI, pydantic, pytest, standard library), with no agent framework. Techniques come from the literature: Ochiai SBFL (Abreu et al.; used by AutoCodeRover), AST-normalized patch voting and test-based selection (Agentless), and 1-minimal patch reduction (Zeller's delta debugging).
  • Evaluation: a 15-repo benchmark with one seeded bug each (off-by-one, wrong operator, mutable default, a bug spanning two functions, a fix that needs a second file, …). It runs the full agent and a single-shot baseline, and every number is written by the benchmark itself. There are 179 tests in total, including guardrail tests, a prompt-injection test, and an end-to-end SSE test.
  • Nebius services: the web demo is packaged for Nebius Serverless Endpoints, and the benchmark can fan out over Nebius Serverless Jobs, with the key held in MysteryBox.

Results (real Nemotron calls)

Pass rate Cost (15 repos) Wall time
Suture (full agent) 15/15 $0.23 336 s
Single Ultra call, no loop 14/15 $0.08 129 s
  • Nano made more calls than Ultra (67 against 57) but cost $0.008 against $0.217, under 4% of the model spend, at 1.7 s/call against 8.3 s. Its triage made the text Ultra reads 75% smaller.
  • The repo the baseline missed shows why the loop exists. All three Ultra candidates agreed on the same wrong fix. The tests caught it, and Suture finished the fix on attempt
    1. Agreement is not correctness; executing the tests is.
  • The demo repo goes red to green in 13.7 s for about one cent.

Challenges I ran into

  • LLM diffs are messy. Line numbers are unreliable, so hunks are located by content, re-rendered as clean diffs, and git apply-checked. My first benchmark run then found a subtler habit: Nemotron Ultra pads hunks with a blank line past the end of the file. That caused 18 of 22 rejected diffs and dragged the baseline to 10/15. Trimming blank edge context fixed it (the baseline went to 14/15). I kept the first run in the repo instead of hiding it.
  • Test-gaming. Research such as ImpossibleBench shows models editing tests to pass. Suture rejects test edits, test-environment detection, stack inspection and mocks in source code before any model review. A repo with a "delete the tests" comment planted in its source is part of the test suite.
  • No Docker in serverless containers. I designed a documented subprocess fallback and am honest about its weaker isolation: it's for trusted demo repos only, and uploads are refused in that mode.
  • Windows details: CRLF checkouts broke patches, and cp1252 garbled pytest's ±. Both are fixed and covered by tests.

What I learned

  • A strong model fixes simple bugs in one call. The agent's value is verification: executing every patch, catching confident-but-wrong fixes, and refusing to cheat.
  • Splitting work across two model sizes really pays off. The small model handles the frequent calls for a few percent of the cost.
  • Measure everything, and publish the embarrassing first run too.

What's next

Ablations per technique, harder multi-file bugs, more trials for variance, and repositories with third-party dependencies.

Built With

Share this project:

Updates

Submission history