Loadbearing flags three public bugs in Haystack and pydantic-ai on the releases that shipped them, and clears the releases that fixed them: 6 of 6 releases called correctly, each on the exact field the issue names. Then it found a fourth bug no one had reported: a regression in the current release of DSPy (3.4.1), found and reproduced, with both releases in the repo as targets.

Judge it in three minutes:

  1. Watch the 3-minute video: https://www.youtube.com/watch?v=EyMuK2FBvmU
  2. Open the live demo: https://loadbearing-demo.vercel.app
  3. Run it yourself, no key, no install beyond Python 3.10: git clone https://github.com/varunikarai/loadbearing.git && cd loadbearing && PYTHONPATH=src python3 -m loadbearing probe examples/ticket_triage/target.json

Inspiration

Pipelines fail silently. They return confident, well-typed output over inputs they are not reading. A retriever renames one field, the prompt builder keeps reading the old one, the model fills the gap with a plausible guess, and every test stays green. In 2024 a tribunal held Air Canada responsible for what its chatbot told a customer (Moffatt v. Air Canada, 2024 BCCRT 149). A confident wrong answer has a cost.

So the question to ask of any pipeline is: which inputs is this system actually load-bearing on? Reading the code gives you an opinion. Loadbearing answers it by running the system.

What it does

  1. Checkpoint once. The pipeline's setup runs once inside a Nebius Token Factory Sandbox and is snapshotted.
  2. Fork per mutation. Every input field is destroyed, dropped and nudged, each in its own fork of that checkpoint, all in parallel. Everything except the one input is byte-identical.
  3. Decide by rule. Two defect classes:
    • Silent passthrough: an input that should matter was destroyed, and the output did not move.
    • Phantom sensitivity: an input that should not matter was nudged, and the output moved.
  4. Explain, fix, prove. Tavily fetches the pipeline's own docs to establish what it is supposed to read. Nemotron 3 Super turns the forks into one cited defect. Nemotron 3 Ultra names the line and writes the smallest fix, and a fresh fork of the same checkpoint proves it.

The demo. A support bot tells a customer: "Yes, you can return the headphones for a full refund within the 30-day return window." The store's policy says 14 days, then warranty only. One command (loadbearing probe ... --backend sandbox --intent --prove) finds that passages is silently ignored, quotes NVIDIA's own page on retrieval-augmented generation as the evidence that it should matter, applies a one-line fix and proves it: the bot now gives the 14-day answer and points to the warranty.

Caught in the wild.

Library The bug Release with it Release with the fix
Haystack split_overlap silently dropped; lived in 37 releases over 15 months 2.30.2: flagged 2.31.0: clean
Haystack AnswerBuilder dropped the caller's meta 2.7.0: flagged 2.8.0: clean
pydantic-ai union NativeOutput ignored the caller's description 2.5.0: flagged 2.5.1: clean
DSPy XMLAdapter never sends a nested output's field descriptions or Literal values to the model 3.4.1: flagged 3.3.0 (last clean): clean

The first three were public issues, replayed. The DSPy one was found by probing the current release; no upstream issue for it existed when we searched.

One command, one verdict, four ways to run it: a CLI whose exit code gates CI, a GitHub Action that fails the build the day a change disconnects an input, loadbearing mcp so a coding agent can probe the pipeline it just wrote before calling the work done, and a hosted demo anyone can open.

How we built it

Sandboxes is the method, not decoration. "Hold everything constant except one input" needs N byte-identical environments. Without checkpoint-and-fork you pay setup N times and still cannot guarantee the environments match, so the comparison is not sound. Sandboxes makes it a primitive. On the demo bot: checkpoint in 4.68 s, then 21 runs in 39.16 s wall instead of 258.71 s one after another (6.6x). On a ticket triage pipeline: 7.6x.

The Nemotron 3 cascade, each model sized to its job:

Piece Job
Nebius Token Factory Sandboxes The checkpoint, and every fork from it
Nemotron 3 Nano One cheap call per moved fork: meaningful change, or noise?
Nemotron 3 Super Reconciles every fork on a field into one defect statement, citing its source
Nemotron 3 Ultra Reads line-numbered source, explains why a path ignores the field, writes the minimal fix
Nebius Token Factory Serves all three models, about 1 s a call
Tavily Fetches the upstream docs that say what a pipeline is supposed to read

Models gather and explain the evidence; a rule makes the call. The verdict is never a model's guess. And every file:line Ultra cites is checked against the file. That check exists because our own recorded demo run cites pipeline.py:18, which is import re; the line it quotes is at 37. A test now runs the check on that recording.

Python, contree-sdk, 236 tests, all offline, CI green on every push, Apache 2.0.

The benchmark, with the test that keeps us honest

54 pipelines with 49 planted input-handling defects:

Detector Cases fully right Defects found Time
Loadbearing (runs it) 50 of 54, identical every run 48 of 49 ~4 s
Nemotron 3 Super reading the code 49 to 51 of 54, changes run to run at temperature 0 45 to 47 ~200 s
Static name check 26 of 54 35 of 49 0 s

Against static analysis the gap is decisive: exact McNemar on cases, p = 1.9e-05. Against a frontier model reading the code it is not significant (p = 0.38), and we publish that. On pipelines this small, reading finds about as much as running. What running adds: the same answer every run, at about 50x the speed, with the forks that justify every verdict and a proven fix. That is what a CI gate needs.

Why it matters

It is for teams putting RAG and agent pipelines into production, and it catches the bug that review and tests miss: an input that quietly stopped mattering. Many coding-agent tools patch code and check that a test passes. Loadbearing asks a different question: what does this system actually depend on? It turns "it looks right" into a check that passes or fails, on every pull request, which is recurring, everyday use of Sandboxes and Nemotron on Token Factory, not a one-time demo.

Challenges we ran into

  • Sandboxes beta access took 12 days, with a bare 403 as the only signal. We built a local backend behind the same interface so work never stopped, then moved the recorded runs onto Sandboxes the day access arrived.
  • LLM pipelines are noisy. We added a noise floor and a confirmation step (a suspicious fork re-runs and counts only if most runs move), so a chance answer is never reported as a defect.
  • Ultra rewrote whole files and touched lines it was never asked about. We switched it to exact old/new edits, so a fix is the smallest possible change.
  • Search found vendor blogs before first-party docs. We made the pipeline's own declared docs the first source, which is why the demo's evidence is NVIDIA's own page.

Accomplishments that we're proud of

  • A regression in DSPy's current release, found and reproduced, and three public library bugs called on 6 of 6 releases.
  • The full loop, find, explain, fix and prove, running on Nebius.
  • A published significance test, including the one that did not go our way.

What we learned

Running a system is stronger evidence than reading it. Models are excellent judges of whether a change is meaningful, but they should not decide what counts as a defect, so in Loadbearing the models gather and explain, and a rule decides.

What's next for Loadbearing

  • Publish the GitHub Action on the Marketplace.
  • Probe multi-step agents, mutating tool results between steps.
  • Scheduled probes across a whole repository with Nebius Serverless Jobs.
  • Fine-tune Nano on Token Factory as a dedicated diff judge.

Team

  • Varunika Rai: the spec; mutations, verdict rules and the noise floor; the Nemotron layer (Nano judging each fork, Super reconciling, Ultra explaining and fixing); Tavily intent inference; the ticket triage target; the pre-public audit; prior art; the visual palette.
  • Aeshvarya Awasthi: the target format and both backends, including the first live Sandboxes runs; fix and prove; the reports; the CLI; the support bot target; the benchmark; CI and the GitHub Action; the hosted demo.

Built With

  • contree
  • github-actions
  • html
  • javascript
  • nebius
  • nebius-sandboxes
  • nebius-token-factory
  • nemotron
  • nemotron-3-nano
  • nemotron-3-super
  • nemotron-3-ultra
  • nvidia
  • python
  • tavily
  • vercel
Share this project:

Updates

Submission history