PatchWarden

An AI agent that turns a production crash alert into a verified, human-reviewed GitHub PR, and proves every fix before it ships.

Comment 1

Inspiration

It's 3:07 a.m. Checkout is down. An AI coding assistant can suggest a fix in seconds, but would you merge it half asleep? It might fix the symptom, break something else, or quietly edit the test so it passes.

We wanted an incident-response agent whose output you can trust without trusting the model. The model proposes; trusted code decides.

What it does

PatchWarden turns a production crash alert into a verified, review-only GitHub pull request:

  1. Alert in. A signed webhook delivers the crash: stack trace, repo, and exact commit.
  2. Fresh sandbox. An E2B sandbox checks out that exact commit. Its network only reaches PyPI.
  3. Reproduce first. If no test ships with the alert, NVIDIA Nemotron writes one from the stack trace. Trusted code accepts it only if it fails at the exact line production reported.
  4. Diagnose. Tavily finds relevant docs (treated as untrusted hints). Nemotron returns file:line evidence, a hypothesis, and its uncertainty.
  5. Patch and verify. Nemotron proposes a minimal fix. It counts as VERIFIED only if the failing test turns green, every original test stays green, and protected test files are byte-for-byte unchanged. A rejected patch is the only thing that moves the run up the model ladder (Nemotron Nano → Super → Ultra).
  6. Challenge the fix. Nemotron may only pick scenarios from a fixed, human-written catalog. Trusted code runs them twice and checks a negative control. In our demo it confirmed a second, unreported defect: one payment key, two charges. With second repair on, it fixes and re-verifies that too.
  7. Human-review PR. The PR contains the fix, a new regression test, the reproduced failure, the second defect, residual risks, Tavily sources, and a hashed evidence manifest. PatchWarden never merges or deploys.

How we built it

  • Models: NVIDIA Nemotron 3 (Nano, Super, Ultra) on Nebius Token Factory through its OpenAI-compatible API, with retries, token caps, a model healthcheck, and a local replay cache so repeated prompts cost no credits.
  • Sandbox: E2B for isolated, per-run checkouts. We started with a local temp-folder adapter behind the same interface, then swapped in E2B.
  • Research: Tavily for cited documentation search.
  • Verifier: pytest red-to-green checks, the full regression suite, SHA-256 hashes of protected tests and contracts, and a patch policy that rejects any diff touching tests, contracts, CI, config, or paths outside the repo.
  • Discovery: a trusted scenario catalog with invariants such as "at most one charge per idempotency key", repeat confirmation, and a known-good negative control.
  • Delivery: FastAPI webhook and run timeline UI, SQLite event store, GitHub API for review-only PRs, hosted on Northflank.

We ask the model for complete corrected files and build the unified diff ourselves with difflib, because model-written diff hunks were unreliable.

Challenges we ran into

  • Our first model test scored 0 of 6, and it was our harness's fault. Nemotron Ultra writes long visible reasoning, so a 1,500-token cap cut replies off, and our parser grabbed the first code block (the quoted original) instead of the last. After fixing both, Nemotron Super fixed the seeded bug 3 of 3 times when it could see the code.
  • Stack traces alone are not enough. With only the trace, the model fixed the crash line but invented functions and broke a passing test (0 of 3). So the agent always reads the files the trace names, and the verifier always runs the full suite, not just the failing test.
  • Avoiding false discoveries. An LLM can call correct behavior a bug. We separated scenario proposal (model) from judgment (human-written invariants run by trusted code), and added a negative control.
  • Small details that break trust, like a file without a final newline producing a diff that git apply rejects.

Accomplishments that we're proud of

  • An end-to-end run from a stack-trace-only alert to a real, review-only GitHub PR in about a minute.
  • A confirmed second defect that no alert, test, or prompt mentioned, with zero false positives in our evaluation.
  • Every number in our evaluation is rebuilt from evidence files.

Evaluation

We ran six controlled cases plus three stack-trace-only alerts, across three Nemotron models and the model ladder. The cases include a known-good negative control that must report no extra defect. Full tables, rebuilt from evidence files, are in docs/evaluation.md.

What we learned

  • Verification, not generation, is the hard and valuable part of AI coding agents.
  • Honest uncertainty states (UNREPRODUCED, PATCH_REJECTED, VERIFY_FAILED) matter as much as success.
  • Measure early: one day-one test changed our whole architecture.

Limitations

The demo app contains seeded bugs. Some logic bugs still need a shipped test. Passing tests are bounded evidence, not proof. PatchWarden never merges or deploys.

What's next

More languages and real-world repos, reproducers built from sanitized request traces, and running the scenario catalog continuously as regression protection.

Built With

+ 1 more
Share this project:

Updates

Submission history