Inspiration
Small teams rarely have a dedicated SRE. When production breaks at night, someone wakes up and spends the first hour collecting logs, reading recent commits and guessing what changed. AI assistants can already suggest fixes, but a confident fix that was never tested is worse than none: it costs review time and erodes trust.
We wanted an on-call agent you could actually trust at 3 a.m., built on one rule: never propose a fix that has not been reproduced and verified.
What it does
Nightwatch receives a production error from Sentry and works through it like a careful engineer:
- Collects evidence - stack trace, request, logs, the recent deploy diff, the failing files and the modules they import, similar past incidents, and known issues of the failing library. Every fragment gets an id like E3 that hypotheses and patches must cite.
- Investigates with NVIDIA Nemotron 3 Super: ranked hypotheses plus a pytest test that should reproduce the failure.
- Reproduces the failure in a throwaway Docker container with no network and no secrets. The test must fail with the same exception, message and location as the alert - and pass on the last good version when it is known.
- Fixes only after a confirmed reproduction, in up to three attempts, escalating to Nemotron 3 Ultra after repeated failures.
- Verifies the patch: the reproduction test and the whole existing suite must pass, and the test files must be byte-identical.
- Hands over a Draft PR with the patch, the test and the evidence - or, when the failure cannot be reproduced (load, timeouts, races, missing data), an honest diagnostic report with facts, hypotheses and next steps instead of an invented fix.
The model only proposes. Code runs the tests, a rule-based verifier decides, a human merges.
How we built it
- Models: Nemotron 3 Super does the work; Ultra takes over after failures; Nano was tested as a cheap triage step. All three run on Nebius Token Factory through one OpenAI-compatible endpoint, with schema-validated JSON replies and cost tracked per call.
- Pipeline (Python): context builder, prompts, patch edits, memory of past incidents, budgets and escalation.
- Sandbox: Docker with no network, a read-only root filesystem, dropped capabilities and resource limits.
- Integrations: a signed Sentry webhook, Draft PR publishing through the GitHub API, a Streamlit operator console.
- Tavily: used only when the error is raised inside a third-party library or the deploy changed a dependency. The query is built by code from the package, its version, the exception type and a scrubbed message - logs, request data and code never leave the service. Results are restricted to GitHub, Stack Overflow, PyPI and official docs; each page becomes separate, citable evidence marked external and unverified, and only pages the model actually relied on appear in the Draft PR as "Known upstream reports". In the final run, for a
packagingregression it returned four GitHub issues about the sameInvalidVersionerror, and the model cited three of them in its hypothesis and patch. - Benchmark: 17 scenarios with known answers - FastAPI endpoints, bugs outside the stack trace, a misleading deploy diff, a library regression, and cases that must end in a report. Each repairable scenario has hidden acceptance tests the model never sees, so we can tell "tests are green" from "the fix is actually right".
Challenges we ran into
Every live failure was traced through the saved prompts and raw model replies and fixed in the pipeline, not in the scenarios:
- Reasoning ate the token budget. With
max_tokens=4096, Super sometimes spent everything on hidden reasoning and returned empty content. We raised the limits, retry cut-off replies with more room, and treat an unusable reply as a failed attempt rather than a crash. - Correct fixes, broken diffs. Super and Ultra wrote the right fix, but their unified diffs invented a context line, so
git applyrejected them. We switched to search/replace edits and build the exact diff in code — tolerant of the shifted indentation models often produce. - Tests that froze the wrong answer. A reproduction test hand-computed an expected output and got it wrong, so the correct fix failed. Reproduction tests now assert that the error is gone plus properties stated in the code or docs.
- The bug was not where it crashed. A consumer rejected bad data built elsewhere, and the producer never reached the model. The context now includes the modules the failing code imports.
- Green is not correct. Some patches passed our tests but missed rarer input forms. That is why we report hidden-test acceptance separately instead of only our own verdict.
On the 7-scenario suite these fixes took us from 4/7 to 7/7 without changing the model.
Accomplishments that we're proud of
Final runs on the 17-scenario suite in the Docker sandbox:
- 17/17 expected outcomes, and 17/17 again on a second run
- 10/12 fixes confirmed by hidden tests the model never saw
- 4/4 correct refusals - no invented fixes for unreproducible failures
- $0.012 and 52 seconds per incident on average
- A working end-to-end path: real Sentry alert → investigation in Docker → Draft PR on GitHub
- 285 automated tests running in CI on every change
We are also proud of reporting the gap honestly: our pipeline and the hidden tests disagree on 2 of 12 fixes, and both numbers are in the README.
What we learned
- With a reasoning model, the engineering around it matters as much as the model: token budgets, output formats and retries decided whether a correct idea became a verified patch.
- Let the model say what to change and let code produce the exact change.
- An agent is only as trustworthy as its verification - and verification needs its own independent check.
- A cheaper first-pass model is not automatically a win: in our A/B run Nano triage added calls and time without improving results, so it stays off by default.
What's next
- Context beyond one import level, and other languages and test frameworks.
- Fine-tuning on verified fixes and their reproduction tests, which Nightwatch already collects.
- Hosted deployment with authentication, so a team can connect Sentry and GitHub in minutes.
- More scenarios and repeated runs to measure variance and the effect of memory properly.
