Inspiration
Cars get crash-test ratings before anyone drives them. Robots whose "brains" are learned policies get a demo video and a shrug. We kept seeing the same thing in robotics: a policy works beautifully in the scenes it was trained on and fails in surprising ways the moment lighting, clutter, or an object changes. Nobody could say where the safe operating envelope ends, because nobody had searched for it.
We wanted to build the crash-test lab that robotics is missing: a system that attacks a robot policy in simulation, thousands of times, with an AI adversary designing the attacks, and hands back a map of exactly where and why it breaks before it ever ships.
What it does
Stress Twin takes a trained manipulation policy and a simulation scene, then runs an adversarial search for failures:
- Scenario design. NVIDIA Nemotron 3 Ultra reads the task and proposes scenario mutations intended to break the robot: lighting, clutter, object color and mass, surface friction, object placement, and a hand entering the workspace.
- Parallel simulation. Nebius Serverless Jobs fan the scenarios out and run them in parallel in [Genesis / MuJoCo], recording video and telemetry for every episode.
- Failure understanding. NVIDIA Nemotron 3 Nano Omni and Cosmos3-Super-Reasoner (hosted on Nebius Token Factory) watch each failure clip and write a plain-language cause, for example "gripper closed early because the object was darker than training data."
- Boundary search. Ultra reads the full results of every round in one context and focuses the next round on the edge between success and failure, so the search converges on the boundary instead of sampling at random.
- The deliverable. A failure map clustered by cause with example clips, a written safety case describing the operating envelope the policy is safe inside, and a retraining dataset built from the discovered failures.
In our test run, Stress Twin evaluated [N] scenarios across [K] rounds and found [M] distinct failure modes in a policy that scored [X]% on its own benchmark.
How we built it
- Reasoning and planning: Nemotron 3 Ultra via the Nebius Token Factory OpenAI-compatible API, with structured JSON output for scenario specs. Its 1M-token context lets one call read an entire round of episode logs.
- Fast everyday calls: Nemotron 3.5 Lightning for log summarization and clustering, to keep latency and credits down.
- Video understanding: Nemotron 3 Nano Omni and Cosmos3-Super-Reasoner on Token Factory for per-episode failure captions.
- Simulation at scale: Nebius Serverless Jobs running containerized [Genesis / MuJoCo] episodes in parallel, built on the Nebius Physical AI workbench workflows.
- Policy under test: [LeRobot ACT / GR00T N1.x] policy trained on [dataset], [link].
- Report and UI: [FastAPI + React / Streamlit] dashboard showing the failure map, clips, and the generated safety case. Deployed on [Nebius Serverless Endpoints / other].
Where Token Factory accelerated the workflow: no GPUs to provision for inference, one API for all three models, and structured outputs that made the scenario specs machine-checkable from day one.
Challenges we ran into
- Making the search adaptive, not random. Random scenario sampling finds the easy failures and misses the boundary. Getting Ultra to reason over whole-round telemetry and propose boundary-focused mutations was the core engineering problem. [Add the concrete thing that broke and how it was fixed.]
- Keeping failure captions honest. Vision models will confidently invent causes. We [cross-check captions against telemetry / require citing a frame range] so every stated cause is grounded in the episode data.
- Simulation determinism. Parallel jobs must reproduce a failure on demand or the map is worthless. [Describe seeding and environment pinning.]
- Scope. With three weeks, we picked one task and one policy and made the whole loop work end to end before adding anything else.
Accomplishments that we're proud of
- A closed loop that goes from policy to failure map to safety case with no human in the middle.
- Failure modes we did not anticipate: [one or two concrete surprising failures, with a sentence each].
- A report format that a safety reviewer can read without knowing anything about machine learning.
What we learned
- Simulation is usually used to train robots. Used to attack them, it answers a question training never does: where does this stop working?
- Long context changes the search strategy. When one model can see an entire round of results, it can reason about the boundary instead of reacting to one episode at a time.
- Judges, regulators, and engineers want the same thing from a safety tool: evidence they can click on.
What's next for Stress Twin
- More simulators and policy formats, starting with Isaac Lab and GR00T.
- Cosmos 3 generation for photoreal domain variations, so discovered sim failures can be stress-tested at the visual level too.
- A closed retraining loop: feed the discovered failures back as training data and re-run Stress Twin until the envelope stops growing.
- A standard report schema so robot safety cases can be compared across vendors, the way crash-test ratings are today.
Log in or sign up for Devpost to join the conversation.