Faultline

The bisect that refuses to blame the innocent. A fresh Nebius Token Factory Sandbox per suspect; NVIDIA Nemotron explains the culprit.

Comment

Inspiration

git bisect is the right algorithm. What is unsafe is running it unattended in the environment you already have. A Python repository pins its dependencies, and the pins move across a commit range, so commit N's test is only right under commit N's pins. Bisect every commit in one reused environment and some verdicts are wrong, the search goes down the wrong half, and an innocent commit gets blamed with total confidence.

We measured it. On a 255-commit history whose pytz pin moves partway through (a fixture we generated, faultbench-tzdata-pipeline), git bisect run in one reused environment named the wrong commit in 10 of 10 runs (two setups, five runs each) — the dependency bump when the environment came from the good ref, a changelog edit when it came from the failing ref. The fixture was built to drift, so those failures are by construction; what it shows is the mechanism.

What it does

You give Faultline a public GitHub repository, a failing test command and the last ref where it passed. It resolves the commit range, suggests the install command from the repository's own packaging files (shown, editable), and starts a hunt.

  • Round 1 tests both ends — the known-good ref and the failing ref — plus as many suspects as the free slots allow, up to eight. Each probe is a disposable Nebius Token Factory Sandbox on the stock python:3.11 image (or, on the fallback backend, a terminating Serverless AI Job): it fetches one commit, installs that commit's declared dependencies from scratch, runs the test and reports.
  • Each round narrows the range — N probes cut it into N+1 segments — and the next round goes out by itself. This deployment holds 16 probe sandboxes at once (the Sandboxes beta allows 50 simultaneous operations). You can close the tab; each probe ticks the hunt when it reports.
  • It ends in a Fault Report or an honest refusal. The report has the guilty commit, its diff, one paragraph from NVIDIA Nemotron 3 Super on Nebius Token Factory explaining why it breaks the test, and a copy-as-GitHub-issue button. The refusals are the failing ref passing in a clean environment (the bug is your environment, not a commit), the known-good ref failing, two contradicting verdicts, or a culprit that hides among commits that will not build. None of them costs an extra job.

The culprit is proven by executed tests. Nemotron only explains it, and the UI labels its paragraph as generated.

How we built it

  • Probes — two backends behind one four-verb ProbeBackend interface (dispatch, status, find, cancel), picked per deploy by FAULTLINE_PROBE_BACKEND. Token Factory Sandboxes (what the live deployment runs): our own TypeScript REST client (POST /v1/instances, GET/DELETE /v1/operations/{id}, GET /v1/operations) spawns one disposable sandbox per commit on the preloaded python:3.11 image with probe.py on stdin, up to 16 at once. Serverless AI Jobs (fallback; cpu-d3 / 2vcpu-8gb, four regions): one terminating job per commit, probe.py delivered through spec.injectedFiles, RS256 service-account auth. Neither needs an image build or a registry.
  • Coordination — Nebius Object Storage (S3 API, conditional writes). No daemon and no database: the hunt record is replaced only with If-Match, and every dispatch is claimed with If-None-Match: "*" first. On the real bucket, 20 races of 2–8 concurrent creators had exactly one winner every time.
  • A probe holds no credential. A public repository's conftest.py is arbitrary code, so a probe gets no bucket mount and no key — not even the Token Factory key that spawned its sandbox — only one presigned URL that can write its own verdict object. That is also why backends and regions are interchangeable: a sandbox, or a uk-south1 job, writes into the eu-west1 bucket over HTTPS.
  • Narration — Nebius Token Factory, nvidia/nemotron-3-super-120b-a12b. One chat-completions call per hunt, after test execution has named the culprit. It is a reasoning model, so the call asks for low reasoning effort with room to answer, reads content only, and treats an empty answer as an error.
  • App — Next.js 15 on Vercel, TypeScript, Tailwind.
  • Tests: 553 unit and integration tests, 9 probe contract tests and 50 end-to-end checks; CI, CodeQL and gitleaks on every push to main and every pull request.

Challenges we ran into

  • Capacity is a parameter of the backend, not a constant. On Jobs it is 2 VMs per region and the third job is rejected, not queued (8 slots across four regions); on Sandboxes it is a configurable slot count under the beta's 50-operation cap (16 here).
  • On Jobs, cold start dominates. A probe's VM takes about 4 min 20 s to start for about 5 s of work, and Nebius throttles job creation (HTTP 429, undocumented), so the engine backs off and re-ticks itself. That is what moved the default to Sandboxes, where a whole probe took 8.6–13.4 s.
  • A job cannot report a result over the Jobs API — no exit code and no logs over REST. We first mounted a read-write bucket into every probe, then replaced it with one presigned PUT per probe, because the thing inside the job is a stranger's test suite.
  • Sandboxes arrived late, with no REST reference we could find. GET /sandboxes/v1/whoami reported every permission false until beta access arrived on 2026-10-14, so the engine was built on Serverless Jobs. The Sandboxes backend was added that day; its request shapes came from reading the Python contree-sdk source, because the app is TypeScript on Vercel. Its operations list has no name filter, so recovering a lost dispatch means matching the probe name we set as the sandbox hostname, and its per-operation cost has no documented unit, so the app shows a Sandboxes hunt's cost as unknown.
  • The account was suspended 21 seconds after signup and stayed that way for 46 days, with no email and an empty-list API that hid the cause. It cleared about a minute after we attached a payment card and topped up $25. The details are in our feedback.

Accomplishments that we're proud of

  • It names the right commit when the baseline cannot. On the drifting history, git bisect run in a reused environment was right 0 of 10 times; Faultline named the planted commit (4e133d6c) in 3 rounds, 19 jobs, 957 s, $0.0948 at list price on Serverless AI Jobs — and in 3 rounds, 20 sandboxes, 46.5 s on Token Factory Sandboxes (evidence/sandboxes-hunt.json). On a real regression — python-attrs/attrs 22.1.0 → 22.2.0, 135 commits — it named 89a890a3, the commit the fix PR itself names (3 rounds, 17 jobs, 919 s).
  • We report where it loses. On Serverless AI Jobs, Faultline is slower than a careful local bisect on both histories (fresh-venv git bisect run: 57 s and 44 s p50). The one Sandboxes hunt (46.5 s) came in under the 57 s, once — a measurement, not a claim.
  • It refuses when there is nothing to blame. Given a test that passes at both ends, a real hunt (074dc7a3) stopped after one round — 8 jobs, 287 s, $0.04 — with "The failing ref passes in a clean environment" and no culprit, instead of inventing one.
  • Infrastructure is never evidence. A job that dies without reporting is retried, then the hunt pauses and says why; a throttled create is waited out. Neither becomes a verdict.
  • Everything is checkable in thirty seconds: /api/integrations/verify names the active probe backend and lists the probe's SHA-256, the latest probe sandboxes read live from the Sandboxes API and the latest probe jobs read live from the Nebius Jobs API; the recorded example on the landing page replays a real hunt with real job IDs.

What we learned

Dependency drift is the quiet failure of unattended bisection, and the answer is not speed but a clean room per commit. On this platform, a batch job is a poor place to hold a credential, and "queued" and "rejected" are very different products. A reasoning model needs an output budget it can spend on the answer, and the model should explain what the tests proved, never decide it.

What's next

A verified fix — Nemotron proposes a patch and one more sandbox runs the failing test on it, so the model proposes and the test decides (not built). More Sandboxes hunts — the attrs regression and repeat runs, so 46.5 s becomes a distribution rather than one number — and a wider band per round now that slots are cheap; Sandboxes checkpoints, to install a commit's dependencies once and branch from them; private repositories; and measuring a repository whose suite takes minutes per commit. We have not measured such a repository, so we do not claim it.

How we used NVIDIA Nemotron, Token Factory and Nebius

  • NVIDIA Nemotron: nvidia/nemotron-3-super-120b-a12b, one call per hunt, with the culprit's diff and the failing test's output; it writes the mechanism, the lines responsible and the fix if there is an obvious one. On the first live hunt it explained that replace(tzinfo=...) skips pytz's zone rules and lands at 06:37 UTC instead of 06:00 (evidence/first-live-hunt.json in the repository). Across the seven hunts that named a culprit the culprit was right every time; the paragraph was right five times and partly wrong twice (DEMO.md §4 and §5), which is why the UI labels it as generated.
  • Nebius Token Factory: the inference endpoint for that call (src/lib/narrate.ts) — no model hosting, no GPU to start or stop for the demo — and Token Factory Sandboxes, where every probe on the live deployment runs (src/lib/backend/sandboxes-backend.ts).
  • Where Token Factory accelerated the workflow: on Sandboxes the same 255-commit hunt went from 923–1,032 s on Serverless AI Jobs to 46.5 s (one run so far), each whole probe taking 8.6–13.4 s; for inference, one OpenAI-compatible call per hunt with no GPU, weights or serving stack of our own.
  • Other Nebius services: Serverless AI Jobs (the selectable fallback probe backend, and every hunt before 2026-10-14), Object Storage (state and per-probe presigned verdicts), IAM service-account token exchange.

Try it out

Testing instructions

  1. Open https://faultline.edycu.dev — no login. The landing page replays a recorded real hunt (255 commits, hunt 827dfe7e, real Nebius job IDs, labelled "recorded run"). It costs nothing to view.
  2. To run a live hunt, fill the three fields under "Start a hunt" (the repository field accepts any public GitHub Python repository; for a known answer use https://github.com/edycutjong/faultbench-tzdata-pipeline, failing test python -m pytest -q tests/test_normalize.py::test_naive_local_midnight_is_interpreted_in_zone, last known good v0.1.0, install pip install -e .[test]; the culprit is 4e133d6c). On the deployment's Token Factory Sandboxes backend that hunt took 46.5 s (one run); on the Serverless AI Jobs fallback it takes 15–17 minutes (if /api/integrations/verify names jobs, the deployment has fallen back). You can close the tab.
  3. The deployment is free to use and shares 16 probe slots, so one live hunt runs at a time. A daily cap protects the credit balance; when it is used up the form shows a "Judge access code" field. If the cap is reached, message us through the Devpost project page for a judge access code.
  4. https://faultline.edycu.dev/api/integrations/verify names the active probe backend and shows the probe's SHA-256, the latest probe sandboxes and the latest probe jobs, read live.
  5. Tests need no credentials: npm install && npm test (README → Getting Started).

Built With

  • github-action
  • github-rest-api
  • nebius-object-storage-(s3-api)
  • nebius-serverless-ai-jobs
  • nebius-token-factory
  • nebius-token-factory-sandbox
  • next.js
  • nvidia-nemotron-3-super
  • playwright
  • python
  • react
  • tailwind-css
  • typescript
  • vercel
  • vitest
Share this project:

Updates

Submission history