Random Bug Walk

Makes web-app bugs the app's own tests miss and added checks catch. Tavily finds sources, Nemotron drafts its spec, and Token Factory models race to build it on a Nebius VM.

Comment

Inspiration

I run evaluations of AI coding agents, and the hardest part has been finding bugs I can trust. A useful bug needs more than a convincing description: I need evidence that it changes the application’s behavior, that the checks detect it, and that a correct fix passes.

I wanted to make bugs for a specific application and keep the ones that hold up. Those cases could help developers find gaps in their tests and give evaluators new tasks for coding agents.

What it does

Random Bug Walk is a prototype bug factory. It creates synthetic bugs that an application’s existing tests miss and separate checks catch.

It currently works with two applications: Umami, an open-source web analytics app, and task-report, a task-and-report service built for experiments. There are three ways to use it:

Build a custom bug. Describe a behavior, or start with the bug designer. The designer uses Tavily to find public bug reports and NVIDIA Nemotron on Nebius Token Factory to turn them into a specification. Review and edit the specification, then start the build. Three Goose agents using Token Factory models race on a Nebius Cloud VM to produce the bug, added checks and a reference fix. The first build that passes qualification is delivered.

Request a catalog case. Choose an application and bug type. The factory checks a prepared case again on fresh application copies and delivers separate packages for the debugging agent and evaluator. Each application currently has one catalog entry.

Inspect or replay the released example. The public results site shows a completed Umami case, its generated report, test results and recorded replays. Judges can also start a fresh replay.

Qualification compares clean, buggy and fixed copies. The application’s original suite must pass on all three, while the added checks must detect the declared failure on the buggy copy and pass on the clean and fixed copies.

Custom builds return an agent bundle, a separate grader bundle and a certificate recording the qualification results, models, timings and metered model spend. The agent bundle contains the buggy application and a symptom-level report; the grader bundle holds the checks and reference fix.

Custom builds cover three previously tested behaviors, plus other non-security behaviors on task-report. Those additional behaviors are labeled “not measured yet,” and their grading rules can be reviewed and edited before building. A request may fail to produce a qualifying bug.

The released Umami example makes the idea concrete. A code rewrite removes a time-zone argument from a pageview query, causing daily counts to fall into UTC days. For Los Angeles, the expected counts are 3, 8, 1; the buggy copy returns 2, 8, 2. All 271 original API tests still pass. Three added time-zone checks detect the problem, while a UTC control passes.

That case passed a 13-copy admission process, including repeated checks and partial-fix and stub-fix probes. Three published replays reproduced the expected results. NVIDIA Nemotron 3.5 Lightning wrote its report from recorded symptoms, and I reviewed it before release.

The results site and published evidence let anyone inspect that example. It covers one pinned application version and one bug shape. Agent difficulty and solve rates remain unmeasured.

How we built it

I directed Claude Code and Codex to write the code. I specified the admission rules, request contract and spending controls, and reviewed the work.

Nebius Token Factory supplies model inference. NVIDIA Nemotron designs and classifies bug specifications. For the released example, Nemotron 3.5 Lightning wrote a pattern card and a user-style report with structured output. Custom builds use Goose agents running other Token Factory models.

Nebius AI Cloud runs the custom-build machine. The web application starts it when needed, and the builder stops it after an idle period. Each agent’s output is qualified on fresh application copies.

Tavily finds source reports for the bug designer. In the released example, it also ran three exact-phrase searches on the generated report. Those searches returned no matching URLs; they check for matching wording, rather than establishing that a bug is novel. Custom builds do not receive that check.

Vercel hosts the Next.js application. Vercel Sandbox runs catalog qualification and released-case replays. Neon Postgres stores the spend ledger and request records.

For the released example, ast-grep rules confirm a public fix’s bug pattern and plant the corresponding synthetic change elsewhere in Umami. Playwright runs the original suite and added checks. Release records and downloadable packages use SHA-256 hashes so their contents can be checked.

The code is public under the MIT license.

Challenges we ran into

Making the checks trustworthy. A failing test can reflect a broken environment or an unrelated error. Clean, buggy and fixed comparisons, repeated checks, and deliberately incomplete fixes helped distinguish the intended failure from those other causes.

Waiting for application copies. In the published candidate attempts, the 13-copy admission took about 45–50 minutes. Writing the report and running the searches took under 20 seconds. Building and testing the application dominated the elapsed time.

Handling uncertain outcomes and costs. A lost connection can leave a job’s state and cost unclear. The candidate and catalog paths reserve spending before work begins and stop for reconciliation when the outcome is uncertain. The custom builder enforces separate model and runtime limits.

Keeping packages reproducible. The application version, container images, checks and reference fix must agree. Pinned inputs and package hashes make those relationships explicit.

Accomplishments that we're proud of

A complete public example connecting a confirmed source pattern, synthetic bug, observed symptom, model-written report, admission results and human-reviewed release.

Three published admissions and three published replays in which Umami’s original API suite passed while the added checks detected the planted defect.

An interactive path from a bug description to competing model builds, qualification results and downloadable packages.

Separate agent and grader bundles, so the evaluator’s checks and reference fix can stay out of the debugging agent’s input.

A working catalog request service that rechecks cases on fresh copies before delivery.

What we learned

A test-suite blind spot and a difficult debugging task are different things. The Umami example demonstrates the former. Its missing argument can still be straightforward to recognize from the source. Measuring difficulty requires separate solver experiments.

Structured model output helps with integration, but it still needs checking. Code can validate a report’s numbers against observations and enforce a specification’s format; neither guarantees that every model judgment is correct.

The application runtime deserves as much attention as model selection. In the published example, rebuilding and testing copies consumed most of the time. Improving that path matters more than shaving a little latency from report generation.

What's next for Random Bug Walk

I want to add more bug shapes, expand the catalogs and connect more host applications. I also want to extend custom behaviors beyond task-report.

The next measurement work is to record complete custom-build times and repeatability, then evaluate coding agents on qualified cases with every submitted fix checked on a fresh copy.

I also plan to connect the public-fix harvest directly to the release pipeline, reducing the prepared input it currently needs, and make the exported cases easier for other evaluators to run.

Built With

Share this project:

Updates

Submission history