Inspiration
In 2024 a tribunal held Air Canada responsible for a refund policy its chatbot made up. Nothing crashed. Nothing threw an error. The answer was fluent, confident and wrong.
That failure has a shape, and it is everywhere: a retriever renames one field, the prompt builder keeps reading the old one, the model fills the gap with a plausible guess, and every test stays green. Today no tool can answer the one question that matters: which of my inputs is this system actually using?
What it does
Loadbearing answers that question by running the pipeline, not reading it.
- Checkpoint once. The pipeline's setup runs once inside a Nebius Token Factory Sandbox and is snapshotted.
- Fork per mutation. Every input field is destroyed, dropped and nudged, each in its own fork of that checkpoint, all in parallel from a byte-identical state.
- Report two defect classes.
- Silent passthrough: an input that should matter was destroyed, and the output did not move.
- Phantom sensitivity: an input that should not matter was nudged, and the output moved.
- Explain, fix, prove. Tavily reads the pipeline's own docs to establish what it is supposed to use. Nemotron 3 Super turns the forks into a cited defect, Nemotron 3 Ultra names the exact line and writes the smallest fix, and the fix is proven in a fresh fork of the same checkpoint.
The demo. A support bot tells a customer: "Yes, you can return the headphones for a full refund within the 30-day return window." The store's policy says 14 days, then warranty only. One command (loadbearing probe ... --backend sandbox --intent --prove) finds that passages is silently ignored, cites NVIDIA's own page on retrieval-augmented generation as the evidence that it should matter, traces the bug to pipeline.py:18, applies a one-line fix and proves it: the bot now gives the 14-day answer and points to the warranty.
It catches real bugs that shipped. We ran public releases of Haystack and pydantic-ai that carry known, reported bugs, alongside the releases that fixed them. Loadbearing called 6 of 6 releases correctly, flagging the exact field each issue names and clearing each fix. One Haystack bug (split_overlap silently dropped) lived in 37 releases over 15 months.
It ships three ways: a CLI, a GitHub Action that fails the build the day a change disconnects an input, and a hosted demo anyone can run.
How we built it
Loadbearing is built on Nebius and NVIDIA because the problem demands it, not as a badge.
| Piece | Its job |
|---|---|
| Nebius Token Factory Sandboxes | The setup run is the checkpoint. Every fork is a run from that snapshot with one mutated input, so the input is the only thing that differs. |
| NVIDIA Nemotron 3 Nano | One cheap call per moved fork: is this change meaningful, or noise? |
| NVIDIA Nemotron 3 Super | Reconciles every fork on a field into one defect statement, citing the source. |
| NVIDIA Nemotron 3 Ultra | Reads the pipeline source, explains why a path ignores the field, and writes the minimal fix. |
| Nebius Token Factory | Serves all three Nemotron models, about 1 s per call. |
| Tavily | Fetches the upstream docs that say what a pipeline is supposed to read. |
Why Sandboxes is essential. "Hold everything constant except one input" means N byte-identical environments. Without checkpoint-and-fork you pay setup N times and still cannot guarantee the environments match, which makes the comparison unsound. Sandboxes turns it into a primitive. On the demo bot: checkpoint in 4.68 s, then 21 runs in 39.16 s instead of 258.71 s (6.6x). On a ticket triage pipeline: 7.6x.
Why the Nemotron 3 cascade. Each model does the job its size is built for. Nano is cheap enough to run once per fork, Super has the reasoning to reconcile evidence, and Ultra handles long-context reasoning over source code. The final verdict is always a rule, never a model's guess, which is what makes the output trustworthy.
Python, contree-sdk, 216 tests, fuzzing of the mutation engine over 3,000 random records, CI on every push, and a demo hosted on Vercel.
The benchmark, stated honestly
54 pipelines with planted input-handling bugs:
| Detector | Cases fully right | Time |
|---|---|---|
| Loadbearing (runs it) | 50 of 54, identical every run | ~4 s |
| Nemotron 3 Super reading the code | 49 to 51 of 54, changes run to run | ~200 s |
| Static name check | 26 of 54 | 0 s |
On small pipelines a frontier model reading the code is about as accurate (the difference is not statistically significant, and we say so). Loadbearing's edge is that it is about 50x faster, gives the same answer every run, shows the forks that justify every verdict, and proves its fix. That is what a CI gate needs. Against static analysis the gap is decisive (exact McNemar, p = 1.9e-05).
Why it matters for Nebius and NVIDIA
- It removes a blocker to production. Teams hesitate to ship RAG and agents because they cannot verify them. Loadbearing turns "it looks right" into a check that passes or fails, which is what gets Nemotron-backed pipelines from prototype to production.
- It is a Sandboxes-native workload. Every probe is dozens of checkpointed sandbox runs plus Nemotron calls, and as a GitHub Action it runs on every pull request. That is recurring, everyday use of Token Factory, not a one-time demo.
- It shows the Nemotron 3 family as a system. Nano, Super and Ultra each doing the job they are sized for, in one pipeline, with measurable results.
- It is open source (Apache 2.0), so any team building on Token Factory can adopt it today.
Challenges we ran into
- Sandboxes beta access took 12 days, with a bare 403 as the only signal. We built a local backend behind the same interface so the work never stopped, then moved every recorded run onto Sandboxes the day access arrived.
- LLM pipelines are noisy. A sampled answer can change on its own. We added a noise floor and a confirmation step (a suspicious fork re-runs and counts only if most runs move), so a chance answer is never reported as a defect.
- Ultra rewrote whole files and touched lines it was never asked about. We switched it to exact old/new edits, so a fix is the smallest possible change.
- Search found vendor blogs before first-party docs. We made the pipeline's own declared docs the first source, which is why the demo's evidence is NVIDIA's own page.
Accomplishments that we're proud of
- Real bugs in real, widely used libraries, called 6 of 6.
- The full loop, find, explain, fix and prove, running on Nebius.
- A tool a team can drop into CI today, with 216 tests behind it.
What we learned
Running a system is a stronger form of evidence than reading it. And models are excellent judges of whether a change is meaningful but should not be the ones deciding what counts as a defect, so in Loadbearing the models gather and explain the evidence and a rule makes the call.
What's next for Loadbearing
- Publish the GitHub Action on the Marketplace.
- Probe multi-step agents, mutating tool results between steps, not only the first input.
- Scheduled probes across a whole repository with Nebius Serverless Jobs.
- Fine-tune Nano on Token Factory as a dedicated diff judge.
Team
- Varunika Rai: the spec; mutations, verdict rules and the noise floor; the Nemotron layer (Nano judging each fork, Super reconciling, Ultra explaining and fixing); Tavily intent inference; the ticket triage target; the pre-public audit; prior art; the visual palette.
- Aeshvarya Awasthi: the target format and both backends, including the first live Sandboxes runs; fix and prove; the reports; the CLI; the support bot target; the benchmark; CI and the GitHub Action; the hosted demo.
Built With
- contree
- github-actions
- html
- javascript
- nebius
- nebius-sandboxes
- nebius-token-factory
- nemotron
- nemotron-3-nano
- nemotron-3-super
- nemotron-3-ultra
- nvidia
- python
- tavily
- vercel
Log in or sign up for Devpost to join the conversation.