Loadbearing flags three public bugs in Haystack and pydantic-ai on the releases that shipped them, and clears the releases that fixed them: 6 of 6 releases called correctly, each on the exact field the issue names. Then it found a fourth bug no one had reported: a regression in the current release of DSPy (3.4.1), found and reproduced, with both releases in the repo as targets.
Judge it in three minutes:
- Watch the 3-minute video: https://www.youtube.com/watch?v=EyMuK2FBvmU
- Open the live demo: https://loadbearing-demo.vercel.app
- Run it yourself, no key, no install beyond Python 3.10:
git clone https://github.com/varunikarai/loadbearing.git && cd loadbearing && PYTHONPATH=src python3 -m loadbearing probe examples/ticket_triage/target.json
Inspiration
Pipelines fail silently. They return confident, well-typed output over inputs they are not reading. A retriever renames one field, the prompt builder keeps reading the old one, the model fills the gap with a plausible guess, and every test stays green. In 2024 a tribunal held Air Canada responsible for what its chatbot told a customer (Moffatt v. Air Canada, 2024 BCCRT 149). A confident wrong answer has a cost.
So the question to ask of any pipeline is: which inputs is this system actually load-bearing on? Reading the code gives you an opinion. Loadbearing answers it by running the system.
What it does
- Checkpoint once. The pipeline's setup runs once inside a Nebius Token Factory Sandbox and is snapshotted.
- Fork per mutation. Every input field is destroyed, dropped and nudged, each in its own fork of that checkpoint, all in parallel. Everything except the one input is byte-identical.
- Decide by rule. Two defect classes:
- Silent passthrough: an input that should matter was destroyed, and the output did not move.
- Phantom sensitivity: an input that should not matter was nudged, and the output moved.
- Explain, fix, prove. Tavily fetches the pipeline's own docs to establish what it is supposed to read. Nemotron 3 Super turns the forks into one cited defect. Nemotron 3 Ultra names the line and writes the smallest fix, and a fresh fork of the same checkpoint proves it.
The demo. A support bot tells a customer: "Yes, you can return the headphones for a full refund within the 30-day return window." The store's policy says 14 days, then warranty only. One command (loadbearing probe ... --backend sandbox --intent --prove) finds that passages is silently ignored, quotes NVIDIA's own page on retrieval-augmented generation as the evidence that it should matter, applies a one-line fix and proves it: the bot now gives the 14-day answer and points to the warranty.
Caught in the wild.
| Library | The bug | Release with it | Release with the fix |
|---|---|---|---|
| Haystack | split_overlap silently dropped; lived in 37 releases over 15 months |
2.30.2: flagged | 2.31.0: clean |
| Haystack | AnswerBuilder dropped the caller's meta |
2.7.0: flagged | 2.8.0: clean |
| pydantic-ai | union NativeOutput ignored the caller's description |
2.5.0: flagged | 2.5.1: clean |
| DSPy | XMLAdapter never sends a nested output's field descriptions or Literal values to the model |
3.4.1: flagged | 3.3.0 (last clean): clean |
The first three were public issues, replayed. The DSPy one was found by probing the current release; no upstream issue for it existed when we searched.
One command, one verdict, four ways to run it: a CLI whose exit code gates CI, a GitHub Action that fails the build the day a change disconnects an input, loadbearing mcp so a coding agent can probe the pipeline it just wrote before calling the work done, and a hosted demo anyone can open.
How we built it
Sandboxes is the method, not decoration. "Hold everything constant except one input" needs N byte-identical environments. Without checkpoint-and-fork you pay setup N times and still cannot guarantee the environments match, so the comparison is not sound. Sandboxes makes it a primitive. On the demo bot: checkpoint in 4.68 s, then 21 runs in 39.16 s wall instead of 258.71 s one after another (6.6x). On a ticket triage pipeline: 7.6x.
The Nemotron 3 cascade, each model sized to its job:
| Piece | Job |
|---|---|
| Nebius Token Factory Sandboxes | The checkpoint, and every fork from it |
| Nemotron 3 Nano | One cheap call per moved fork: meaningful change, or noise? |
| Nemotron 3 Super | Reconciles every fork on a field into one defect statement, citing its source |
| Nemotron 3 Ultra | Reads line-numbered source, explains why a path ignores the field, writes the minimal fix |
| Nebius Token Factory | Serves all three models, about 1 s a call |
| Tavily | Fetches the upstream docs that say what a pipeline is supposed to read |
Models gather and explain the evidence; a rule makes the call. The verdict is never a model's guess. And every file:line Ultra cites is checked against the file. That check exists because our own recorded demo run cites pipeline.py:18, which is import re; the line it quotes is at 37. A test now runs the check on that recording.
Python, contree-sdk, 236 tests, all offline, CI green on every push, Apache 2.0.
The benchmark, with the test that keeps us honest
54 pipelines with 49 planted input-handling defects:
| Detector | Cases fully right | Defects found | Time |
|---|---|---|---|
| Loadbearing (runs it) | 50 of 54, identical every run | 48 of 49 | ~4 s |
| Nemotron 3 Super reading the code | 49 to 51 of 54, changes run to run at temperature 0 | 45 to 47 | ~200 s |
| Static name check | 26 of 54 | 35 of 49 | 0 s |
Against static analysis the gap is decisive: exact McNemar on cases, p = 1.9e-05. Against a frontier model reading the code it is not significant (p = 0.38), and we publish that. On pipelines this small, reading finds about as much as running. What running adds: the same answer every run, at about 50x the speed, with the forks that justify every verdict and a proven fix. That is what a CI gate needs.
Why it matters
It is for teams putting RAG and agent pipelines into production, and it catches the bug that review and tests miss: an input that quietly stopped mattering. Many coding-agent tools patch code and check that a test passes. Loadbearing asks a different question: what does this system actually depend on? It turns "it looks right" into a check that passes or fails, on every pull request, which is recurring, everyday use of Sandboxes and Nemotron on Token Factory, not a one-time demo.
Challenges we ran into
- Sandboxes beta access took 12 days, with a bare 403 as the only signal. We built a local backend behind the same interface so work never stopped, then moved the recorded runs onto Sandboxes the day access arrived.
- LLM pipelines are noisy. We added a noise floor and a confirmation step (a suspicious fork re-runs and counts only if most runs move), so a chance answer is never reported as a defect.
- Ultra rewrote whole files and touched lines it was never asked about. We switched it to exact old/new edits, so a fix is the smallest possible change.
- Search found vendor blogs before first-party docs. We made the pipeline's own declared docs the first source, which is why the demo's evidence is NVIDIA's own page.
Accomplishments that we're proud of
- A regression in DSPy's current release, found and reproduced, and three public library bugs called on 6 of 6 releases.
- The full loop, find, explain, fix and prove, running on Nebius.
- A published significance test, including the one that did not go our way.
What we learned
Running a system is stronger evidence than reading it. Models are excellent judges of whether a change is meaningful, but they should not decide what counts as a defect, so in Loadbearing the models gather and explain, and a rule decides.
What's next for Loadbearing
- Publish the GitHub Action on the Marketplace.
- Probe multi-step agents, mutating tool results between steps.
- Scheduled probes across a whole repository with Nebius Serverless Jobs.
- Fine-tune Nano on Token Factory as a dedicated diff judge.
Team
- Varunika Rai: the spec; mutations, verdict rules and the noise floor; the Nemotron layer (Nano judging each fork, Super reconciling, Ultra explaining and fixing); Tavily intent inference; the ticket triage target; the pre-public audit; prior art; the visual palette.
- Aeshvarya Awasthi: the target format and both backends, including the first live Sandboxes runs; fix and prove; the reports; the CLI; the support bot target; the benchmark; CI and the GitHub Action; the hosted demo.
Built With
- contree
- github-actions
- html
- javascript
- nebius
- nebius-sandboxes
- nebius-token-factory
- nemotron
- nemotron-3-nano
- nemotron-3-super
- nemotron-3-ultra
- nvidia
- python
- tavily
- vercel