FlipWitness

• 

  FlipWitness: a test is a witness only when it fails before the patch and passes after it.

• 

  From bug report to reproducible test: Nemotron 3 Nano extracts, Ultra hypothesises, Super writes tests.

• 

  Live run on the month-end sample: four witnesses, nothing failing after the patch, verdict supported.

• 

  An incomplete patch: tests still fail after it, so the evidence card says patch suspect.

• 

  Every round is shown: rewrites run only when there is no witness, up to three times.

• 

  Every recorded live run is published, misses included.

The problem

"The tests pass" says little about a bug fix. Tests that pass on the old code and on the new code show nothing about the bug. A test that fails after the fix says the patch broke something. What a reviewer wants is a test that fails on the original code and passes on the patched code: a witness that the patch fixes the reported behaviour.

Writing such a test by hand takes time, and a model asked to "write tests for this fix" has no way to tell whether its tests prove anything. FlipWitness makes the before/after rule the measure of success and lets the model iterate until it meets it, or says plainly that it did not.

What it does

Pick a sample case (three small bugs, each with a good patch and an incomplete patch) or paste your own small module, a bug report and a unified diff. FlipWitness:

1. extracts the expected behaviour, the actual behaviour, the reproduction steps and the inputs from the report;
2. proposes one to three root-cause hypotheses tied to real files, each with edge cases worth testing;
3. writes candidate tests;
4. runs every test twice in your browser, on the original code and on the patched code;
5. classifies each test: witness (fails before, passes after), passes both (not evidence), fails after (the patch is suspect) or inconclusive (error or timeout);
6. if there is no witness, feeds the before/after results back and asks for new tests, up to three rewrites.

The result is an evidence card: supported (at least one witness and nothing failing after the patch), patch suspect (some test fails after the patch) or no evidence, with the per-test before/after table, the test source of every round and the cost of the session.

How it works

• Nemotron 3 Nano reads the bug report and returns the expected/actual/repro fields as JSON.
• Nemotron 3 Ultra reads the extraction, the code and the patch and returns root-cause hypotheses. Code drops any hypothesis that names a file that does not exist.
• Nemotron 3 Super writes the tests against a tiny harness (test, expect(...).toBe/.toEqual/.toThrow/.toBeCloseTo). On a rewrite it sees which tests passed on both sides, which failed after the patch and why.
• The tests never run on the server. They run in a dedicated Web Worker in your browser. Network APIs (fetch, XMLHttpRequest, WebSocket, EventSource), nested workers and dynamic imports are disabled, and a run that loops is stopped after 1.5 seconds. Before that, a static check rejects test code that imports anything other than the module under test and the harness, or that uses process, eval, Function, globalThis and similar tokens. The check is a filter, not a security boundary; the Worker is the isolation, and 14 attempts to reach the network or the host from a test were all stopped in headless Chrome.
• All three models are called through the Nebius Token Factory API with structured JSON output. The app's server side is a serverless function that makes those calls; a shared ledger caps spending per session, per day and in total, and when a cap is reached the app shows the recorded sample results instead.

Recorded results

Each sample was run live several times on 2026-10-24 (expected: good patch → supported, incomplete patch → patch suspect):

The sample results shown in the app are selected runs, and the README lists every run. In the shipped csv-line run with the good patch, three generated tests are witnesses, but three others expect wrong values and fail after the patch, so the card says "patch suspect" and shows exactly which tests and why. A session costs between $0.005 and $0.023.

Challenges

• Code inside strict JSON. With reasoning on and a strict JSON schema, the generated test source never contained a newline, a double quote or a backslash (10 of 10 recorded). A // comment then swallowed the rest of the line, and CSV inputs lost their quotes. The test prompt now asks for single quotes, String.fromCharCode(34) for a double quote and no comments; a looser JSON mode kept the escapes but produced runaway answers.
• Reasoning inside the token budget. Nano returns its reasoning in a separate reasoning field and its usage report has no reasoning breakdown; at a 700-token limit, the JSON answer was cut after 70 characters. Stage limits are now 3,000 tokens for Nano, 6,000 for Ultra and 8,000 for Super.
• Latency. Recorded calls took from 2.6 to 30.5 seconds (one test-writing probe took 54.5 seconds), so the page shows each stage's model and progress, and the server allows 55 seconds per call.

What's next

• Support for more languages than JavaScript modules.
• A GitHub pull-request mode that posts the evidence card as a review comment.
• A larger set of real bug fixes to measure how often the loop finds a witness.

Built With

• css
• html
• javascript
• nebius-token-factory
• node.js
• nvidia-nemotron
• upstash-redis
• vercel
• web-workers